spike(embed): F-04 BGE-M3 dense+sparse — NO-GO

Reason: SentenceTransformer.encode() in sentence-transformers 5.5.1 does
not accept return_dense / return_sparse kwargs. Those belong to the
FlagEmbedding.BGEM3FlagModel API, which is not in our dependency set.

Dense encoding works fine via the vanilla encode() call (shape (N, 1024),
float32). Sparse / lexical_weights requires either:
  (a) adding FlagEmbedding as a dependency, or
  (b) using sentence_transformers.SparseEncoder with a SPLADE checkpoint, or
  (c) driving the underlying transformers model and sparse head manually.

Branch retained as documentation per docs/loops/spike-gate.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

.claude/README.md

@@ -1,21 +0,0 @@
-# .claude/ — Harness-Config für codex-py
-
-`settings.json` ist **committed** (team-weit): jede Worker-Session in diesem Repo
-bekommt dieselben Permissions. Persönliche Abweichungen gehören in
-`settings.local.json` (gitignored).
-
-## Was die Permissions tun
-- **`defaultMode: acceptEdits`** — Edits/Writes ohne Prompt (Worker schreiben Code
-  in ihrem F-XX-Paket frei). Disjunkte Ownership je Paket (siehe knowledge-base
-  `STATE.md`); Konflikte löst der Leader bei der Integration.
-- **allow** — die Dev-Toolchain (`uv`, `ruff`, `mypy`, `pytest`, `python`), git
-  read+workflow, `podman` (lokale pgvector-DB), `ollama` (lokales LLM), `curl`
-  (API-Tests, z. B. OpenAlex) und gängige Read-Utils.
-- **deny** — `git push --force`/`-f`, `git clean -fd(x)`, `sudo rm`, katastrophale
-  `rm -rf /`/`~`. Hart blockiert. Alles übrige (z. B. `git reset --hard`) ist
-  *nicht* allow-gelistet → es fragt nach.
-
-## Hinweis
-Gespiegelt am Stil der knowledge-base-`.claude/`, aber auf den Python/uv-Stack
-zugeschnitten. Tests laufen via `uv run pytest`; falls `uv run` die Umgebung
-neu-synct, ist `.venv/bin/python` ebenfalls allow-gelistet.

.claude/settings.json

@@ -1,80 +0,0 @@
-{
-  "$schema": "https://json.schemastore.org/claude-code-settings.json",
-  "env": {
-    "CLAUDE_VERBOSE": "1"
-  },
-  "outputStyle": "Explanatory",
-  "permissions": {
-    "defaultMode": "acceptEdits",
-    "allow": [
-      "Bash(uv:*)",
-      "Bash(ruff:*)",
-      "Bash(mypy:*)",
-      "Bash(pytest:*)",
-      "Bash(python -m pytest:*)",
-      "Bash(python3 -m pytest:*)",
-      "Bash(python:*)",
-      "Bash(python3:*)",
-      "Bash(.venv/bin/python:*)",
-      "Bash(podman:*)",
-      "Bash(ollama:*)",
-      "Bash(codespell:*)",
-      "Bash(shellcheck:*)",
-      "Bash(git status:*)",
-      "Bash(git diff:*)",
-      "Bash(git log:*)",
-      "Bash(git show:*)",
-      "Bash(git branch:*)",
-      "Bash(git ls-remote:*)",
-      "Bash(git ls-files:*)",
-      "Bash(git remote:*)",
-      "Bash(git rev-parse:*)",
-      "Bash(git rev-list:*)",
-      "Bash(git merge-base:*)",
-      "Bash(git diff-tree:*)",
-      "Bash(git describe:*)",
-      "Bash(git tag:*)",
-      "Bash(git add:*)",
-      "Bash(git commit:*)",
-      "Bash(git switch:*)",
-      "Bash(git checkout:*)",
-      "Bash(git restore:*)",
-      "Bash(git stash:*)",
-      "Bash(git fetch:*)",
-      "Bash(git pull:*)",
-      "Bash(git push:*)",
-      "Bash(git merge:*)",
-      "Bash(git rebase:*)",
-      "Bash(git worktree:*)",
-      "Bash(git init:*)",
-      "Bash(git mv:*)",
-      "Bash(ls:*)",
-      "Bash(find:*)",
-      "Bash(grep:*)",
-      "Bash(rg:*)",
-      "Bash(wc:*)",
-      "Bash(cat:*)",
-      "Bash(head:*)",
-      "Bash(tail:*)",
-      "Bash(awk:*)",
-      "Bash(jq:*)",
-      "Bash(curl:*)",
-      "Bash(mkdir:*)",
-      "Bash(cp:*)",
-      "Bash(test:*)",
-      "Bash(echo:*)",
-      "Bash(true)"
-    ],
-    "deny": [
-      "Bash(git push --force:*)",
-      "Bash(git push -f:*)",
-      "Bash(git clean -fdx:*)",
-      "Bash(git clean -fd:*)",
-      "Bash(rm -rf /)",
-      "Bash(rm -rf /*)",
-      "Bash(rm -rf ~)",
-      "Bash(rm -rf ~/*)",
-      "Bash(sudo rm:*)"
-    ]
-  }
-}

.env.example

@@ -5,12 +5,6 @@
 # Example: postgresql://researcher:change_me@localhost:5432/papers
 DATABASE_URL=postgresql://researcher:change_me@localhost:5432/papers
 
-# Optional: privileged connection used by `codex migrate` to apply schema DDL.
-# The app DATABASE_URL is often a least-privilege DML role that cannot CREATE/
-# ALTER owner-held tables; point this at an owner/superuser connection.
-# Falls back to DATABASE_URL when unset.
-# MIGRATION_DATABASE_URL=postgresql://postgres:change_me@localhost:5432/papers
-
 # GROBID service base URL (containerised — see infra/docker-compose.yml)
 GROBID_URL=http://localhost:8070
 
@@ -29,12 +23,3 @@ EMBEDDING_DIM=1024
 # E-mail address for the OpenAlex Polite Pool (faster rate limits).
 # Required by OpenAlex ToS when making automated requests.
 OPENALEX_MAILTO=you@example.com
-
-# Nougat HTTP-Server (Jetson: http://jetson.local:8080 | lokal: http://localhost:8080)
-# Used by `codex ingest <id> --rich` for full-PDF Mathpix Markdown output.
-NOUGAT_URL=http://localhost:8080
-
-# F-16 Chunk Quality Gate thresholds (all optional — defaults shown)
-CHUNK_MIN_CHARS=60          # Discard chunks shorter than this many characters
-CHUNK_MIN_ALPHA_RATIO=0.40  # Discard chunks with < 40% alphabetic characters
-CHUNK_MAX_BIB_SCORE=0.70    # Discard chunks scoring above this bibliography threshold

.gitignore

@@ -18,8 +18,6 @@ build/
 
 # Environment secrets — NEVER commit
 .env
-.env.*
-!.env.example
 
 # Type-checker and linter caches
 .mypy_cache/
@@ -35,9 +33,3 @@ htmlcov/
 .idea/
 .vscode/
 *.swp
-
-# Claude Code — settings.json IS committed (team-wide); only personal overrides ignored
-.claude/settings.local.json
-
-# Generated wiki index (regenerated by `codex wiki compile`; concepts.yaml is the source)
-wiki/index.md

README.md

@@ -19,11 +19,13 @@ $EDITOR .env
 # 3. Install Python dependencies (requires uv)
 uv sync
 
-# 4. Apply the database schema (idempotent — safe to re-run after upgrades)
+# 4. Apply the database schema (first run only)
-#    Needs a role that can run DDL. If DATABASE_URL is a least-privilege app
+uv run python -c "
-#    role, point MIGRATION_DATABASE_URL at an owner/superuser connection first:
+from codex.db import get_conn, apply_schema
-#      MIGRATION_DATABASE_URL=postgresql://postgres:...@host:5432/papers
+with get_conn() as conn:
-uv run codex migrate
+    apply_schema(conn)
+print('Schema applied.')
+"
 ```
 
 ---
@@ -32,8 +34,7 @@ uv run codex migrate
 
 | Variable | Default | Description |
 |---|---|---|
-| `DATABASE_URL` | `postgresql://researcher:change_me@localhost:5432/papers` | libpq connection string (app role; DML) |
+| `DATABASE_URL` | `postgresql://researcher:change_me@localhost:5432/papers` | libpq connection string |
-| `MIGRATION_DATABASE_URL` | *(falls back to `DATABASE_URL`)* | Privileged connection used by `codex migrate` for schema DDL (owner/superuser) |
 | `GROBID_URL` | `http://localhost:8070` | GROBID HTTP API base URL |
 | `OLLAMA_BASE_URL` | `http://localhost:11434` | Local Ollama endpoint (optional) |
 | `EMBEDDING_MODEL` | `BAAI/bge-m3` | sentence-transformers model name |
@@ -80,8 +81,8 @@ Maps C++ symbols (qualified names or `file.cpp:line` references) to the
 papers they implement.  The workflow:
 
 1. Tag C++ source with `@cite <bibkey>` in Doxygen comments.
-2. Run `codex provenance scan --lib-path <path>` to scan and resolve tags.
+2. Run `codex provenance sync --lib-path <path>` to scan and resolve tags.
-3. Run `codex provenance bib <out.bib>` to generate a `.bib` file
+3. Run `codex provenance export-bib <out.bib>` to generate a `.bib` file
    containing **only the cited subset** of your collection.
 
 The exported `.bib` is a derived view of the master catalogue — regenerate
@@ -96,79 +97,13 @@ codex ingest <id>               # ingest one paper by arXiv ID or DOI
 codex ingest-file <ids.txt>     # bulk ingest from a file of IDs
 codex search "<query>" [--hybrid]
 codex discover leads
-codex provenance scan --lib-path <path>
+codex provenance sync --lib-path <path>
-codex provenance bib <out.bib>
+codex provenance export-bib <out.bib>
 codex ask "<question>"          # optional LLM Q&A via Ollama
 ```
 
 ---
 
-## MCP Server (F-14)
-
-codex exposes its KB as read-only MCP tools for LLM clients (Claude Code, Claude Desktop, etc.).
-
-### Start (stdio)
-
-```bash
-# directly
-python -m codex.mcp_server
-
-# or via the installed script
-codex-mcp
-```
-
-### Client registration
-
-**Claude Code** — add to `.mcp.json` in your project root:
-
-```json
-{
-  "mcpServers": {
-    "codex": {
-      "command": "uv",
-      "args": ["run", "codex-mcp"],
-      "cwd": "/path/to/codex-py"
-    }
-  }
-}
-```
-
-**Claude Desktop** — add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
-
-```json
-{
-  "mcpServers": {
-    "codex": {
-      "command": "uv",
-      "args": ["--directory", "/path/to/codex-py", "run", "codex-mcp"]
-    }
-  }
-}
-```
-
-### HTTP transport (optional, config-gated)
-
-```bash
-MCP_TRANSPORT=http MCP_AUTH_TOKEN=<secret> codex-mcp
-```
-
-HTTP transport requires `MCP_AUTH_TOKEN` — the server refuses to start without it.
-Requests must carry `Authorization: Bearer <token>`.
-
-### Available tools (read-only)
-
-| Tool | Description |
-|---|---|
-| `search` | Chunk-level hybrid search (dense BGE-M3) |
-| `ask` | RAG answer (returns "not available" until implemented) |
-| `wiki_read` | Compiled concept page (F-12) |
-| `wiki_list` | All concept pages + freshness |
-| `discover_leads` | Dangling-citation leads |
-| `provenance_verify` | Scan C++ sources for @cite tags |
-| `synthesis_browse` | Leads/conjectures from leads/ (F-13) |
-
----
-
 ## Development
 
 ```bash

codex/__main__.py

@@ -1,3 +0,0 @@
-from codex.cli import app
-
-app()

codex/cli.py

@@ -1,726 +1,8 @@
-"""codex CLI — commands wired to domain modules."""
-
-from __future__ import annotations
-
-import json
-from datetime import UTC, datetime
-from typing import Optional
-
 import typer
 
 app = typer.Typer(help="codex — personal knowledge base for scientific papers.")
-discover_app = typer.Typer(help="Discovery queries over the citation graph.")
-prov_app = typer.Typer(help="Provenance: @cite scan, code_links, bib export.")
-wiki_app = typer.Typer(help="Wiki-compile: grounded concept pages over the RAG substrate.")
-synth_app = typer.Typer(help="Synthesis: grounded leads and quarantined conjectures (F-13).")
-quality_app = typer.Typer(help="Chunk quality gate and section classification (F-16).")
-graph_app = typer.Typer(help="Citation graph analytics: PageRank, coupling, dangling leads (F-15).")
-# F-09: search sub-app with paper (semantic) and formula (FTS) subcommands
-search_app = typer.Typer(
-    help="Search: semantic paper search and formula full-text search.",
-    invoke_without_command=True,
-)
-app.add_typer(discover_app, name="discover")
-app.add_typer(prov_app, name="provenance")
-app.add_typer(wiki_app, name="wiki")
-app.add_typer(synth_app, name="synthesis")
-app.add_typer(quality_app, name="quality")
-app.add_typer(graph_app, name="graph")
-app.add_typer(search_app, name="search")
 
 
-@app.command()
+@app.callback()
-def migrate() -> None:
+def _main() -> None:
-    """Apply infra/schema.sql to bring the database schema up to date (idempotent).
+    """codex CLI — commands land here in F-07."""
-
-    Must connect as a role that can run DDL (owns / can CREATE + ALTER the tables).
-    The application's DATABASE_URL is usually a least-privilege DML role and will
-    fail with InsufficientPrivilege; set MIGRATION_DATABASE_URL to a privileged
-    (owner/superuser) connection (audit M-1).
-    """
-    import psycopg
-    import psycopg.rows
-
-    from codex.config import get_settings
-    from codex.db import apply_schema
-
-    settings = get_settings()
-    url = (settings.migration_database_url or settings.database_url).get_secret_value()
-    try:
-        with psycopg.connect(url, row_factory=psycopg.rows.dict_row) as conn:
-            apply_schema(conn)
-    except psycopg.errors.InsufficientPrivilege as exc:
-        typer.echo(
-            "ERROR: schema migration needs a role that owns the tables "
-            "(CREATE/ALTER). Set MIGRATION_DATABASE_URL to a privileged connection "
-            f"and retry. ({exc})",
-            err=True,
-        )
-        raise typer.Exit(1) from exc
-    typer.echo("Schema applied — database is up to date.")
-
-
-@app.command()
-def ingest(
-    paper_id: str = typer.Argument(..., help="arXiv ID, DOI, or OpenAlex W-ID"),
-    source: Optional[str] = typer.Option(None, "--source", "-s", help="Path to .tex or .pdf"),  # noqa: UP045
-    rich: bool = typer.Option(  # noqa: A002
-        False,
-        "--rich",
-        help="Extract formulas and figures (requires PDF source, F-09).",
-    ),
-) -> None:
-    """Ingest a paper into the knowledge base."""
-    from codex.ingest import ingest_paper
-
-    result = ingest_paper(paper_id, source_path=source, rich=rich)
-    typer.echo(
-        f"Ingested {result.paper_id}: {result.chunks_upserted} chunks, "
-        f"{result.citations_upserted} citations, "
-        f"{result.formulas_upserted} formulas, "
-        f"{result.figures_upserted} figures"
-    )
-
-
-# ---------------------------------------------------------------------------
-# F-09: Search command group
-# ---------------------------------------------------------------------------
-
-
-@search_app.callback(invoke_without_command=True)
-def search_callback(ctx: typer.Context) -> None:
-    """Search subcommands: ``paper`` for semantic search, ``formula`` for LaTeX FTS."""
-    if ctx.invoked_subcommand is None:
-        typer.echo(ctx.get_help())
-        raise typer.Exit(0)
-
-
-@search_app.command("paper")
-def search_paper(
-    query: str = typer.Argument(..., help="Natural-language search query"),
-    limit: int = typer.Option(10, "--limit", "-n", help="Number of results"),
-    cite_boost: bool = typer.Option(
-        False,
-        "--cite-boost",
-        help="Re-rank the top results by citation PageRank — a within-page "
-        "tie-breaker, not a hard re-ranking (F-15). Graceful when corpus < 5 papers.",
-    ),
-) -> None:
-    """Semantic similarity search over paper abstracts."""
-    from codex.config import get_settings
-    from codex.db import get_conn
-    from codex.embed import get_embedder
-
-    emb = get_embedder().encode_dense([query])[0].tolist()
-    with get_conn() as conn:
-        rows = conn.execute(
-            """
-            SELECT id, title, year,
-                   abstract_emb <-> %(emb)s::vector AS distance
-            FROM   papers
-            WHERE  abstract_emb IS NOT NULL
-            ORDER  BY abstract_emb <-> %(emb)s::vector
-            LIMIT  %(limit)s
-            """,
-            {"emb": emb, "limit": limit},
-        ).fetchall()
-
-        if cite_boost and rows:
-            from codex.graph import build_citation_graph, citation_pagerank
-
-            settings = get_settings()
-            graph = build_citation_graph(conn)
-            pr = citation_pagerank(graph)
-            alpha = settings.graph_cite_boost_alpha
-            results = []
-            for row in rows:
-                pr_score = pr.get(row["id"], 0.0)
-                # distance is lower=better; divide to reduce distance for high-PR papers
-                boosted = row["distance"] / (1.0 + alpha * pr_score)
-                results.append((boosted, row))
-            results.sort(key=lambda x: x[0])
-            for boosted_dist, row in results:
-                typer.echo(f"[{boosted_dist:.3f}] {row['id']} ({row['year']}) — {row['title']}")
-            return
-
-    for row in rows:
-        typer.echo(f"[{row['distance']:.3f}] {row['id']} ({row['year']}) — {row['title']}")
-
-
-@search_app.command("formula")
-def search_formula(
-    query: str = typer.Argument(..., help="LaTeX snippet or keyword to search in formulas"),
-    limit: int = typer.Option(10, "--limit", "-n", help="Number of results"),
-) -> None:
-    """Full-text search over extracted LaTeX formulas (F-09)."""
-    from codex.db import get_conn
-
-    with get_conn() as conn:
-        rows = conn.execute(
-            """
-            SELECT paper_id, page, raw_latex, context,
-                   ts_rank(to_tsvector('english', raw_latex || ' ' || coalesce(context, '')),
-                            plainto_tsquery('english', %(query)s)) AS rank
-            FROM   formulas
-            WHERE  to_tsvector('english', raw_latex || ' ' || coalesce(context, ''))
-                   @@ plainto_tsquery('english', %(query)s)
-            ORDER  BY rank DESC
-            LIMIT  %(limit)s
-            """,
-            {"query": query, "limit": limit},
-        ).fetchall()
-    if not rows:
-        typer.echo("No matching formulas found.")
-        return
-    for row in rows:
-        typer.echo(
-            f"[{row['rank']:.3f}] {row['paper_id']} p.{row['page']}  {row['raw_latex'][:80]}"
-        )
-
-
-@discover_app.command("leads")
-def discover_leads(
-    limit: int = typer.Option(20, "--limit", "-n"),
-) -> None:
-    """Show discovery leads: cited papers not yet ingested, ranked by pull."""
-    from codex.discover import discovery_leads
-
-    for item in discovery_leads(limit=limit):
-        typer.echo(f"{item['pull']:>4}×  {item['cited_id']}")
-
-
-@discover_app.command("recommend")
-def discover_recommend(
-    paper_id: str = typer.Argument(
-        ..., help="Semantic Scholar paper ID (or arXiv:/DOI: prefixed)."
-    ),
-    limit: int = typer.Option(20, "--limit", "-n", help="Number of recommendations."),
-) -> None:
-    """Recommended papers for PAPER_ID via Semantic Scholar."""
-    from codex.sources.semanticscholar import fetch_recommendations
-
-    rec = fetch_recommendations(paper_id, limit=limit)
-    if not rec:
-        typer.echo(f"No recommendations found for {paper_id}.")
-        return
-    for pid in rec:
-        typer.echo(pid)
-
-
-@discover_app.command("citing")
-def discover_citing(paper_id: str = typer.Argument(...)) -> None:
-    """List papers that cite PAPER_ID."""
-    from codex.discover import citing_papers
-
-    for pid in citing_papers(paper_id):
-        typer.echo(pid)
-
-
-@discover_app.command("cited-by")
-def discover_cited_by(paper_id: str = typer.Argument(...)) -> None:
-    """List papers that PAPER_ID cites."""
-    from codex.discover import cited_by
-
-    for pid in cited_by(paper_id):
-        typer.echo(pid)
-
-
-@discover_app.command("cocited")
-def discover_cocited(
-    paper_id: str = typer.Argument(...),
-    limit: int = typer.Option(10, "--limit", "-n"),
-) -> None:
-    """List papers frequently co-cited with PAPER_ID."""
-    from codex.discover import cocited_papers
-
-    for item in cocited_papers(paper_id, limit=limit):
-        typer.echo(f"{item['co_citations']:>4}×  {item['paper_id']}")
-
-
-@prov_app.command("scan")
-def prov_scan(root_dir: str = typer.Argument(..., help="Root directory to scan")) -> None:
-    """Scan C++ sources for @cite tags."""
-    from codex.provenance import scan_cite_tags
-
-    hits = scan_cite_tags(root_dir)
-    typer.echo(json.dumps(hits, indent=2))
-
-
-@prov_app.command("add-link")
-def prov_add_link(
-    symbol: str = typer.Argument(...),
-    paper_id: Optional[str] = typer.Option(None, "--paper-id"),  # noqa: UP045
-    role: Optional[str] = typer.Option(None, "--role"),  # noqa: UP045
-    note: Optional[str] = typer.Option(None, "--note"),  # noqa: UP045
-) -> None:
-    """Add a code → paper link."""
-    from codex.provenance import add_code_link
-
-    link = add_code_link(symbol, paper_id=paper_id, role=role, note=note)
-    typer.echo(f"Created code_link id={link.id}: {symbol} → {paper_id}")
-
-
-@prov_app.command("links")
-def prov_links(
-    paper_id: Optional[str] = typer.Option(None, "--paper-id"),  # noqa: UP045
-) -> None:
-    """List code → paper links."""
-    from codex.provenance import list_code_links
-
-    for link in list_code_links(paper_id=paper_id):
-        typer.echo(f"[{link.id}] {link.symbol} → {link.paper_id}  ({link.role})")
-
-
-@prov_app.command("bib")
-def prov_bib(
-    paper_ids: Optional[list[str]] = typer.Argument(None),  # noqa: UP045, B008
-    output: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--output", "-o", help="Write to file instead of stdout"
-    ),
-) -> None:
-    """Export papers as BibTeX."""
-    from codex.provenance import export_bib
-
-    bib = export_bib(paper_ids if paper_ids else None)
-    if output:
-        from pathlib import Path
-
-        Path(output).write_text(bib, encoding="utf-8")
-        typer.echo(f"Wrote {output}")
-    else:
-        typer.echo(bib)
-
-
-@app.command()
-def ask(question: str = typer.Argument(..., help="Question to answer")) -> None:
-    """Ask a question (RAG — not yet implemented)."""
-    typer.echo("ask: not yet implemented", err=True)
-    raise typer.Exit(1)
-
-
-# ---------------------------------------------------------------------------
-# F-12: Wiki command group
-# ---------------------------------------------------------------------------
-
-
-@wiki_app.command("compile")
-def wiki_compile(
-    concept: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--concept", help="Compile only this concept slug."
-    ),
-    all_concepts: bool = typer.Option(
-        False, "--all", help="Force full recompile (ignore change detection)."
-    ),
-    top_k: int = typer.Option(0, "--top-k", help="Override config.wiki_top_k (0 = use config)."),
-    output_dir: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--output-dir", help="Override config.wiki_dir."
-    ),
-) -> None:
-    """Compile concept pages from retrieved chunks via local LLM.
-
-    By default only recompiles concepts whose source chunks have changed
-    (hash-based incremental). Use --all to force full recompile.
-    """
-    from codex.wiki import compile_all
-
-    report = compile_all(
-        changed_only=not all_concepts,
-        top_k=top_k if top_k > 0 else None,
-        concept_filter=concept,
-        output_dir=output_dir,
-    )
-
-    if report.compiled:
-        typer.echo(f"Compiled: {', '.join(report.compiled)}")
-    if report.skipped:
-        typer.echo(f"Skipped (unchanged): {', '.join(report.skipped)}")
-    if report.ungrounded:
-        typer.echo(f"⚠ Ungrounded claims: {len(report.ungrounded)}", err=True)
-        for slug, text in report.ungrounded:
-            typer.echo(f"  [{slug}] {text[:100]}", err=True)
-
-
-@wiki_app.command("list")
-def wiki_list(
-    output_dir: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--output-dir", help="Override config.wiki_dir."
-    ),
-) -> None:
-    """List compiled concept pages with freshness information."""
-    import json as _json
-    from pathlib import Path
-
-    from codex.config import get_settings
-
-    settings = get_settings()
-    wiki_dir = Path(output_dir or settings.wiki_dir)
-    state_path = wiki_dir / ".compile-state.json"
-
-    state: dict[str, str] = {}
-    if state_path.exists():
-        try:
-            state = _json.loads(state_path.read_text(encoding="utf-8"))
-        except (_json.JSONDecodeError, OSError):
-            state = {}
-
-    pages = sorted(wiki_dir.glob("*.md"))
-    if not pages:
-        typer.echo("No compiled pages found.")
-        return
-
-    for page in pages:
-        if page.name in ("index.md", "log.md"):
-            continue
-        slug = page.stem
-        hash_val = state.get(slug, "—")[:8]
-        content = page.read_text(encoding="utf-8")
-        n_claims = content.count("[")
-        n_ungrounded = content.count("⚠")
-        mtime = datetime.fromtimestamp(page.stat().st_mtime, tz=UTC).strftime("%Y-%m-%d %H:%M")
-        typer.echo(
-            f"{slug:40s}  last={mtime}  hash={hash_val}  claims≈{n_claims}  ⚠={n_ungrounded}"
-        )
-
-
-@wiki_app.command("check")
-def wiki_check(
-    output_dir: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--output-dir", help="Override config.wiki_dir."
-    ),
-) -> None:
-    """Check all compiled pages for ungrounded claims.
-
-    Exits with code 0 if all claims are grounded, 1 if any are ungrounded.
-    Suitable for CI pipelines.
-    """
-    from pathlib import Path
-
-    from codex.config import get_settings
-
-    settings = get_settings()
-    wiki_dir = Path(output_dir or settings.wiki_dir)
-
-    pages = sorted(wiki_dir.glob("*.md"))
-    found_ungrounded = False
-    for page in pages:
-        if page.name in ("index.md", "log.md"):
-            continue
-        content = page.read_text(encoding="utf-8")
-        if "⚠" in content:
-            typer.echo(f"⚠ Ungrounded claim(s) in: {page.name}", err=True)
-            found_ungrounded = True
-
-    # Check for quarantined pages in wiki/draft/
-    draft_dir = wiki_dir / "draft"
-    draft_pages = sorted(draft_dir.glob("*.md")) if draft_dir.exists() else []
-    if draft_pages:
-        n = len(draft_pages)
-        typer.echo(
-            f"⚠ {n} page(s) quarantined in wiki/draft/ (grounding rate below threshold)",
-            err=True,
-        )
-        for dp in draft_pages:
-            typer.echo(f"  - {dp.name}", err=True)
-        raise typer.Exit(2)
-
-    if found_ungrounded:
-        raise typer.Exit(1)
-    else:
-        typer.echo("All claims grounded.")
-
-
-# ---------------------------------------------------------------------------
-# F-13: Synthesis command group
-# ---------------------------------------------------------------------------
-
-
-@synth_app.command("leads")
-def synthesis_leads(
-    lib_path: Optional[str] = typer.Option(  # noqa: UP045
-        None,
-        "--lib-path",
-        help=(
-            "Path to a C++ source tree. Enables improvement leads (@cite-aware) and "
-            "code-vs-corpus gap detection."
-        ),
-    ),
-    topic: list[str] | None = typer.Option(  # noqa: B008
-        None,
-        "--topic",
-        help=(
-            "Topic to check for coverage gaps (repeatable). Opt-in: without --topic "
-            "no coverage-map gaps run (avoids per-paper false positives, audit R-8)."
-        ),
-    ),
-    output_dir: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--output-dir", help="Override config.leads_dir."
-    ),
-) -> None:
-    """Generate grounded leads (stages 2-4) and write them to leads/grounded/."""
-    from codex.config import get_settings
-    from codex.synthesis import (
-        default_llm_client,
-        find_connections,
-        find_gaps,
-        find_improvements,
-        write_leads,
-    )
-
-    settings = get_settings()
-    leads_dir = output_dir or settings.leads_dir
-    llm = default_llm_client()
-
-    connections = find_connections(top_k=settings.synthesis_top_k, llm=llm)
-    gaps = find_gaps(lib_path=lib_path, llm=llm, topics=topic or None)
-    improvements = (
-        find_improvements(lib_path, top_k=settings.synthesis_top_k, llm=llm) if lib_path else []
-    )
-
-    all_leads = connections + gaps + improvements
-    write_leads(all_leads, leads_dir)
-
-    typer.echo(
-        f"Grounded leads written to {leads_dir}/grounded/: "
-        f"{len(connections)} connection · {len(gaps)} gap · {len(improvements)} improvement"
-    )
-
-
-@synth_app.command("conjectures")
-def synthesis_conjectures(
-    output_dir: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--output-dir", help="Override config.leads_dir."
-    ),
-) -> None:
-    """Generate stage-5 conjecture leads — quarantined to leads/conjectures/.
-
-    INVARIANT (F-13 HARD): conjectures are NEVER written to wiki/. They
-    require a ``suggested_validation`` spike before they may be promoted.
-    """
-    from codex.config import get_settings
-    from codex.synthesis import default_llm_client, propose_conjectures, write_leads
-
-    settings = get_settings()
-    leads_dir = output_dir or settings.leads_dir
-    llm = default_llm_client()
-
-    conjectures = propose_conjectures(top_k=settings.synthesis_top_k, llm=llm)
-    write_leads(conjectures, leads_dir)
-
-    typer.echo(f"Conjectures (UNVERIFIED) written to {leads_dir}/conjectures/: {len(conjectures)}")
-
-
-@synth_app.command("report")
-def synthesis_report(
-    output_dir: Optional[str] = typer.Option(  # noqa: UP045
-        None, "--output-dir", help="Override config.leads_dir."
-    ),
-) -> None:
-    """Show grounded leads and conjectures — visibly separated."""
-    from pathlib import Path
-
-    from codex.config import get_settings
-
-    settings = get_settings()
-    leads_dir = Path(output_dir or settings.leads_dir)
-    grounded_dir = leads_dir / "grounded"
-    conj_dir = leads_dir / "conjectures"
-
-    grounded_files = sorted(grounded_dir.glob("*.md")) if grounded_dir.exists() else []
-    conj_files = sorted(conj_dir.glob("*.md")) if conj_dir.exists() else []
-
-    typer.echo("=" * 72)
-    typer.echo(f"GROUNDED LEADS  ({len(grounded_files)})  → {grounded_dir}")
-    typer.echo("=" * 72)
-    if not grounded_files:
-        typer.echo("(none)")
-    for f in grounded_files:
-        first = f.read_text(encoding="utf-8").splitlines()[0]
-        typer.echo(f"  {first}")
-
-    typer.echo("")
-    typer.echo("=" * 72)
-    typer.echo(f"CONJECTURES  ({len(conj_files)})  → {conj_dir}  [UNVERIFIED — never in wiki/]")
-    typer.echo("=" * 72)
-    if not conj_files:
-        typer.echo("(none)")
-    for f in conj_files:
-        first = f.read_text(encoding="utf-8").splitlines()[0]
-        typer.echo(f"  {first}")
-
-
-# ---------------------------------------------------------------------------
-# F-16 — quality sub-commands
-# ---------------------------------------------------------------------------
-
-
-@quality_app.command("run")
-def quality_run(
-    paper_id: Optional[str] = typer.Option(  # noqa: UP045
-        None,
-        "--paper-id",
-        help="Restrict pass to one paper (omit to process all papers).",
-    ),
-) -> None:
-    """Apply quality gate + section classification to existing chunks.
-
-    Iterates over chunks in the database, removes those that fail the
-    quality thresholds, and back-fills the ``section`` column for the rest.
-    """
-    from codex.config import get_settings
-    from codex.db import get_conn
-    from codex.quality import run_quality_pass
-
-    settings = get_settings()
-    with get_conn() as conn:
-        stats = run_quality_pass(paper_id=paper_id, conn=conn, settings=settings)
-
-    scope = f"paper {paper_id}" if paper_id else "all papers"
-    typer.echo(
-        f"Quality pass ({scope}): "
-        f"{stats['kept']} kept, "
-        f"{stats['removed']} removed, "
-        f"{stats['tagged']} section-tagged"
-    )
-
-
-# ---------------------------------------------------------------------------
-# F-15 — graph sub-commands
-# ---------------------------------------------------------------------------
-
-
-@graph_app.command("report")
-def graph_report(
-    top_n: int = typer.Option(10, "--top-n", "-n", help="Number of hub papers to show."),
-    output_json: bool = typer.Option(False, "--json", help="Output as JSON."),
-) -> None:
-    """Show citation graph report: top hub papers, dangling citations, cluster summary."""
-    from codex.config import get_settings
-    from codex.db import get_conn
-    from codex.graph import (
-        build_citation_graph,
-        citation_pagerank,
-        citing_coverage,
-        dangling_citations,
-    )
-
-    settings = get_settings()
-    with get_conn() as conn:
-        graph = build_citation_graph(conn)
-        paper_rows = conn.execute("SELECT id, title FROM papers").fetchall()
-    known_ids = {row["id"] for row in paper_rows}
-    titles = {row["id"]: row["title"] for row in paper_rows}
-
-    if graph.number_of_nodes() == 0:
-        typer.echo("Citation graph is empty — ingest some papers first.")
-        return
-
-    n_papers = len(known_ids)
-    pr = citation_pagerank(graph)
-    hubs = sorted(pr.items(), key=lambda x: -x[1])[:top_n]
-    dangling = sorted(dangling_citations(graph, known_ids))
-    n_citing, _ = citing_coverage(graph, known_ids)
-    coverage = n_citing / n_papers if n_papers else 0.0
-    small_corpus = n_papers < settings.graph_min_corpus_size
-    warning = (
-        f"only {n_papers} ingested papers; recommended >= {settings.graph_min_corpus_size} "
-        "for meaningful ranking"
-        if small_corpus
-        else None
-    )
-
-    if output_json:
-        typer.echo(
-            json.dumps(
-                {
-                    "nodes": graph.number_of_nodes(),
-                    "edges": graph.number_of_edges(),
-                    "citing_coverage": {
-                        "citing": n_citing,
-                        "papers": n_papers,
-                        "fraction": round(coverage, 4),
-                    },
-                    "small_corpus_warning": warning,  # null unless below threshold (audit U-3)
-                    "hubs": [
-                        {"paper_id": pid, "title": titles.get(pid), "pagerank": score}
-                        for pid, score in hubs
-                    ],
-                    "dangling_count": len(dangling),
-                    "dangling": dangling,
-                },
-                indent=2,
-            )
-        )
-        return
-
-    if warning:
-        typer.echo(f"Warning: {warning}.", err=True)
-    # R-D: low citing-paper coverage starves PageRank/coupling even at a healthy
-    # paper count — warn on the share of papers that actually have out-edges.
-    if coverage < settings.graph_min_citing_coverage:
-        typer.echo(
-            f"Warning: only {n_citing}/{n_papers} papers ({coverage:.0%}) have out-edges "
-            f"(recommended ≥ {settings.graph_min_citing_coverage:.0%}); "
-            f"low citing coverage weakens PageRank/coupling.",
-            err=True,
-        )
-    typer.echo(f"Graph: {graph.number_of_nodes()} nodes, {graph.number_of_edges()} edges")
-    typer.echo(f"Citing coverage: {n_citing}/{n_papers} papers with out-edges ({coverage:.0%})")
-    typer.echo("")
-    typer.echo(f"TOP-{top_n} HUB PAPERS (PageRank)")
-    typer.echo("-" * 48)
-    for pid, score in hubs:
-        # in-KB hubs show their title; dangling (OpenAlex-id) hubs are flagged (U-1)
-        label = titles.get(pid) or "(dangling — not ingested)"
-        typer.echo(f"  {score:.4f}  {pid}  {label}")
-    typer.echo("")
-    typer.echo(f"DANGLING CITATIONS ({len(dangling)} cited but not ingested)")
-    typer.echo("-" * 48)
-    for pid in dangling[:top_n]:
-        typer.echo(f"  {pid}")
-    if len(dangling) > top_n:
-        typer.echo(f"  … and {len(dangling) - top_n} more")
-
-
-@graph_app.command("related")
-def graph_related(
-    paper_id: str = typer.Argument(..., help="Paper ID to find related papers for."),
-    min_shared: int = typer.Option(
-        2, "--min-shared", help="Minimum shared references for bibliographic coupling."
-    ),
-) -> None:
-    """List bibliographically related papers (shared references ≥ min-shared)."""
-    from codex.db import get_conn
-    from codex.graph import build_citation_graph, find_related
-
-    with get_conn() as conn:
-        graph = build_citation_graph(conn)
-
-    related = find_related(paper_id, graph, min_shared=min_shared)
-    if not related:
-        typer.echo(f"No papers with ≥ {min_shared} shared references found for {paper_id}.")
-        return
-    typer.echo(
-        f"Papers related to {paper_id} (bibliographic coupling, ≥ {min_shared} shared refs):"
-    )
-    for pid in related:
-        typer.echo(f"  {pid}")
-
-
-@graph_app.command("cocited")
-def graph_cocited(
-    paper_id: str = typer.Argument(..., help="Paper ID to find co-cited papers for."),
-) -> None:
-    """List papers frequently co-cited with PAPER_ID (graph co-citation)."""
-    from codex.db import get_conn
-    from codex.graph import build_citation_graph, find_co_cited
-
-    with get_conn() as conn:
-        graph = build_citation_graph(conn)
-
-    results = find_co_cited(paper_id, graph)
-    if not results:
-        typer.echo(f"No papers co-cited with {paper_id}.")
-        return
-    typer.echo(f"Papers co-cited with {paper_id} (count desc):")
-    for pid, count in results:
-        typer.echo(f"  {count:>3}×  {pid}")

codex/config.py

@@ -9,7 +9,7 @@ from __future__ import annotations
 
 from functools import lru_cache
 
-from pydantic import AliasChoices, Field, SecretStr
+from pydantic import Field
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 
@@ -26,25 +26,14 @@ class Settings(BaseSettings):
     # ------------------------------------------------------------------
     # Database
     # ------------------------------------------------------------------
-    database_url: SecretStr = Field(
+    database_url: str = Field(
-        default=SecretStr("postgresql://researcher:change_me@localhost:5432/papers"),
+        default="postgresql://researcher:change_me@localhost:5432/papers",
         description=(
             "libpq-compatible connection string consumed by psycopg. "
             "Example: postgresql://user:pass@host:5432/dbname"
         ),
     )
 
-    migration_database_url: SecretStr | None = Field(
-        default=None,
-        description=(
-            "Optional privileged connection string used by `codex migrate` to apply "
-            "schema DDL. The app's database_url is typically a least-privilege DML "
-            "role that cannot CREATE/ALTER owner-held tables (raises "
-            "InsufficientPrivilege); point this at an owner/superuser connection. "
-            "Falls back to database_url when unset."
-        ),
-    )
-
     # ------------------------------------------------------------------
     # External services
     # ------------------------------------------------------------------
@@ -53,12 +42,6 @@ class Settings(BaseSettings):
         description="Base URL of the GROBID HTTP API (containerised).",
     )
 
-    nougat_url: str = Field(
-        default="http://localhost:8080",
-        validation_alias=AliasChoices("NOUGAT_URL", "nougat_url"),
-        description="Base URL of the Nougat OCR HTTP API (containerised).",
-    )
-
     ollama_base_url: str = Field(
         default="http://localhost:11434",
         description="Base URL of the local Ollama endpoint (optional Q&A layer).",
@@ -95,245 +78,6 @@ class Settings(BaseSettings):
         ),
     )
 
-    # ------------------------------------------------------------------
-    # F-12 Wiki-Compile
-    # ------------------------------------------------------------------
-    wiki_dir: str = Field(
-        default="wiki/",
-        description=(
-            "Directory where compiled wiki pages are written. "
-            "Relative paths are resolved from the current working directory."
-        ),
-    )
-
-    wiki_llm_model: str = Field(
-        default="qwen2.5:7b",
-        description=(
-            "Ollama model name used for wiki synthesis. "
-            "Must be available at the configured Ollama endpoint (WIKI_LLM_URL or "
-            "OLLAMA_BASE_URL). Default: qwen2.5:7b (qwen-light profile on Jetson)."
-        ),
-    )
-
-    wiki_llm_url: str | None = Field(
-        default=None,
-        description=(
-            "Ollama base URL for wiki synthesis. "
-            "When None, falls back to OLLAMA_BASE_URL "
-            "(http://jetson.local:11434 for the Jetson). "
-            "Set WIKI_LLM_URL to override."
-        ),
-    )
-
-    wiki_top_k: int = Field(
-        default=12,
-        gt=0,
-        description=(
-            "Number of top chunks retrieved per concept for wiki synthesis. "
-            "Higher values improve recall at the cost of a larger LLM prompt."
-        ),
-    )
-
-    wiki_min_grounding_rate: float = Field(
-        default=0.5,
-        ge=0.0,
-        le=1.0,
-        description=(
-            "Minimum fraction of claims that must be grounded for a page to be written "
-            "to wiki/<slug>.md. Pages below this threshold are quarantined to "
-            "wiki/draft/<slug>.md instead."
-        ),
-    )
-
-    # ------------------------------------------------------------------
-    # F-09 Rich Parsing
-    # ------------------------------------------------------------------
-    mathpix_app_id: SecretStr | None = Field(
-        default=None,
-        description=(
-            "MathPix App ID for cloud formula extraction. "
-            "When None, pix2tex local OCR is used as fallback."
-        ),
-    )
-
-    mathpix_app_key: SecretStr | None = Field(
-        default=None,
-        description=(
-            "MathPix App Key for cloud formula extraction. "
-            "Must be set together with MATHPIX_APP_ID."
-        ),
-    )
-
-    pix2tex_fallback: bool = Field(
-        default=True,
-        description=(
-            "Enable pix2tex (local LaTeX OCR) as fallback when MathPix credentials "
-            "are absent. Set to False to disable formula extraction entirely without creds."
-        ),
-    )
-
-    figures_dir: str = Field(
-        default="figures/",
-        description=(
-            "Directory where extracted figure images are written. "
-            "Relative paths are resolved from the current working directory."
-        ),
-    )
-
-    # ------------------------------------------------------------------
-    # F-13 Synthesis / Lead-Engine
-    # ------------------------------------------------------------------
-    leads_dir: str = Field(
-        default="leads/",
-        description=(
-            "Directory where synthesis leads are written. "
-            "Grounded leads land in <leads_dir>/grounded/, conjectures in "
-            "<leads_dir>/conjectures/. INVARIANT: conjectures NEVER touch wiki/."
-        ),
-    )
-
-    synthesis_llm_model: str = Field(
-        default="qwen2.5:7b",
-        description=(
-            "Ollama model name used for synthesis (connection / gap / improvement / "
-            "conjecture). Defaults to the qwen-light profile on the Jetson."
-        ),
-    )
-
-    synthesis_llm_url: str | None = Field(
-        default=None,
-        description=("Ollama base URL for synthesis. When None, falls back to OLLAMA_BASE_URL."),
-    )
-
-    synthesis_top_k: int = Field(
-        default=20,
-        gt=0,
-        description=(
-            "Number of top chunks retrieved per synthesis probe. Higher values "
-            "improve recall at the cost of a larger LLM prompt."
-        ),
-    )
-
-    synthesis_min_grounded_ratio: float = Field(
-        default=0.5,
-        ge=0.0,
-        le=1.0,
-        description=(
-            "Minimum fraction of claims that must be grounded for a grounded lead "
-            "(connection / gap / improvement) to be emitted. Leads below this "
-            "threshold are dropped rather than written to leads/grounded/."
-        ),
-    )
-
-    synthesis_gap_min_coverage: int = Field(
-        default=2,
-        gt=0,
-        description=(
-            "A topic is considered a 'gap' candidate when it is covered by fewer "
-            "than this many distinct bibkeys in the retrieved top-K."
-        ),
-    )
-
-    # ------------------------------------------------------------------
-    # F-14 MCP Server
-    # ------------------------------------------------------------------
-    mcp_transport: str = Field(
-        default="stdio",
-        description=(
-            "Transport protocol for the MCP server. "
-            "Allowed values: 'stdio' (default) or 'http'. "
-            "HTTP transport requires MCP_AUTH_TOKEN to be set."
-        ),
-    )
-
-    mcp_host: str = Field(
-        default="127.0.0.1",
-        description="Bind host for HTTP transport (ignored for stdio).",
-    )
-
-    mcp_port: int = Field(
-        default=8765,
-        gt=0,
-        description="Bind port for HTTP transport (ignored for stdio).",
-    )
-
-    mcp_auth_token: SecretStr | None = Field(
-        default=None,
-        description=(
-            "Bearer token required when mcp_transport='http'. "
-            "If HTTP transport is requested but this is None, the server refuses to start."
-        ),
-    )
-
-    # ------------------------------------------------------------------
-    # F-16 Chunk Quality Gate
-    # ------------------------------------------------------------------
-    chunk_min_chars: int = Field(
-        default=60,
-        gt=0,
-        description=(
-            "Minimum character count for a chunk to pass the quality gate. "
-            "Chunks shorter than this value are discarded before embedding."
-        ),
-    )
-
-    chunk_min_alpha_ratio: float = Field(
-        default=0.40,
-        ge=0.0,
-        le=1.0,
-        description=(
-            "Minimum fraction of alphabetic characters in a chunk. "
-            "Chunks with a lower ratio are treated as OCR artefacts and discarded."
-        ),
-    )
-
-    chunk_max_bib_score: float = Field(
-        default=0.70,
-        ge=0.0,
-        le=1.0,
-        description=(
-            "Maximum bibliography heuristic score before a chunk is rejected. "
-            "Score is based on DOI density, 'et al.' and year-in-brackets patterns."
-        ),
-    )
-
-    # ------------------------------------------------------------------
-    # F-15 Literature Graph
-    # ------------------------------------------------------------------
-    graph_cite_boost_alpha: float = Field(
-        default=0.3,
-        ge=0.0,
-        le=1.0,
-        description=(
-            "Weight of the PageRank citation-boost in hybrid search. "
-            "boosted_distance = distance / (1 + alpha * pagerank_score) — dividing "
-            "lowers the cosine distance of high-PageRank papers (lower = closer). "
-            "Set to 0.0 to disable the boost without removing the flag."
-        ),
-    )
-
-    graph_min_corpus_size: int = Field(
-        default=15,
-        gt=0,
-        description=(
-            "Minimum number of ingested papers for citation_pagerank to yield "
-            "meaningful rankings. Below this threshold a warning is logged and "
-            "uniform scores are returned."
-        ),
-    )
-
-    graph_min_citing_coverage: float = Field(
-        default=0.8,
-        ge=0.0,
-        le=1.0,
-        description=(
-            "Minimum share of ingested papers that must have ≥1 out-edge (i.e. "
-            "cite something) before `codex graph report` warns. Low citing "
-            "coverage starves PageRank/coupling even when the paper count looks "
-            "healthy — the share of *citing* papers is what matters (R-D)."
-        ),
-    )
-
 
 @lru_cache(maxsize=1)
 def get_settings() -> Settings:

codex/db.py

@@ -36,7 +36,7 @@ def get_conn() -> Generator[psycopg.Connection[psycopg.rows.DictRow], None, None
     """
     settings = get_settings()
     with psycopg.connect(
-        settings.database_url.get_secret_value(),
+        settings.database_url,
         row_factory=psycopg.rows.dict_row,
     ) as conn:
         yield conn
@@ -45,16 +45,11 @@ def get_conn() -> Generator[psycopg.Connection[psycopg.rows.DictRow], None, None
 def apply_schema(conn: psycopg.Connection[psycopg.rows.DictRow]) -> None:
     """Idempotently apply ``infra/schema.sql`` to the connected database.
 
-    Safe to call repeatedly — every statement is ``CREATE … IF NOT EXISTS`` or
+    Safe to call on every startup — all DDL statements use
-    ``ADD COLUMN IF NOT EXISTS``, so re-application is a no-op.
+    ``CREATE … IF NOT EXISTS``.
-
-    Requires a connection whose role can run DDL (owns / can CREATE + ALTER the
-    tables). The least-privilege application role is typically DML-only and will
-    raise ``InsufficientPrivilege``; use a privileged connection — see
-    ``codex migrate`` and ``MIGRATION_DATABASE_URL`` (audit M-1).
 
     Args:
-        conn: An open psycopg connection with DDL privileges.
+        conn: An open psycopg connection (obtained via :func:`get_conn`).
     """
     schema_path = Path(__file__).parent.parent / "infra" / "schema.sql"
     sql = schema_path.read_text(encoding="utf-8")

codex/discover.py

@@ -1,81 +0,0 @@
-"""Discovery queries over the citation graph stored in PostgreSQL.
-
-Every query resolves ``citations.cited_id`` to a canonical ``papers.id`` through
-the shared :data:`codex.graph.RESOLVED_CITATIONS_SQL`, so this module and the
-citation-graph layer agree on what counts as "ingested". Previously this module
-compared the raw OpenAlex ``cited_id`` against ``papers.id`` (a DOI/arXiv id),
-which mis-reported already-ingested papers as discovery leads (audit C-1).
-"""
-
-from __future__ import annotations
-
-from codex.db import get_conn
-from codex.graph import RESOLVED_CITATIONS_SQL
-
-
-def discovery_leads(limit: int = 20) -> list[dict[str, int | str]]:
-    """Return papers referenced by already-ingested papers but not yet collected.
-
-    Returns list of dicts with keys:
-        cited_id (str)   — canonical id of the referenced (not-yet-ingested) work
-        pull     (int)   — how many already-ingested papers cite this target
-
-    Ordered by pull DESC, limited to `limit` rows. References that resolve to a
-    paper already in the corpus are excluded — they are not leads.
-    """
-    sql = f"""
-        WITH resolved AS ({RESOLVED_CITATIONS_SQL})
-        SELECT cited_id, count(*) AS pull
-        FROM   resolved
-        WHERE  cited_id NOT IN (SELECT id FROM papers)
-        GROUP  BY cited_id
-        ORDER  BY pull DESC
-        LIMIT  %(limit)s
-    """
-    with get_conn() as conn:
-        rows = conn.execute(sql, {"limit": limit}).fetchall()
-    return [{"cited_id": row["cited_id"], "pull": row["pull"]} for row in rows]
-
-
-def citing_papers(paper_id: str) -> list[str]:
-    """Return IDs of all papers that cite `paper_id` (in-graph reverse citations)."""
-    sql = f"""
-        WITH resolved AS ({RESOLVED_CITATIONS_SQL})
-        SELECT citing_id FROM resolved WHERE cited_id = %(paper_id)s
-    """
-    with get_conn() as conn:
-        rows = conn.execute(sql, {"paper_id": paper_id}).fetchall()
-    return [row["citing_id"] for row in rows]
-
-
-def cited_by(paper_id: str) -> list[str]:
-    """Return IDs of all papers that `paper_id` cites (forward citations)."""
-    sql = f"""
-        WITH resolved AS ({RESOLVED_CITATIONS_SQL})
-        SELECT cited_id FROM resolved WHERE citing_id = %(paper_id)s
-    """
-    with get_conn() as conn:
-        rows = conn.execute(sql, {"paper_id": paper_id}).fetchall()
-    return [row["cited_id"] for row in rows]
-
-
-def cocited_papers(paper_id: str, limit: int = 10) -> list[dict[str, int | str]]:
-    """Return papers frequently co-cited with `paper_id`.
-
-    A paper X is co-cited with `paper_id` if some paper cites both.
-    Returns list of dicts: {paper_id: str, co_citations: int}, ordered DESC.
-    """
-    sql = f"""
-        WITH resolved AS ({RESOLVED_CITATIONS_SQL})
-        SELECT c2.cited_id AS paper_id, count(*) AS co_citations
-        FROM   resolved c1
-        JOIN   resolved c2 ON c1.citing_id = c2.citing_id
-        WHERE  c1.cited_id = %(paper_id)s
-          AND  c2.cited_id != %(paper_id)s
-        GROUP  BY c2.cited_id
-        ORDER  BY co_citations DESC
-        LIMIT  %(limit)s
-    """
-    with get_conn() as conn:
-        rows = conn.execute(sql, {"paper_id": paper_id, "limit": limit}).fetchall()
-    return [{"paper_id": row["paper_id"], "co_citations": row["co_citations"]} for row in rows]

codex/embed.py

@@ -1,160 +0,0 @@
-"""Dense (+ optional sparse) embeddings via BGE-M3 (ADR-0002).
-
-Uses :class:`FlagEmbedding.BGEM3FlagModel` for encoding because the
-``return_dense`` / ``return_sparse`` kwargs are part of the FlagEmbedding
-API — not ``sentence-transformers``.  The dense output is identical to a
-vanilla ``SentenceTransformer`` load of ``BAAI/bge-m3`` (same weights,
-same model); sparse output is a list of ``{token_id: weight}`` dicts per
-text.
-
-Status: only the **dense** path is wired into ingest/search today; the search
-"hybrid" is dense + Postgres FTS. :meth:`Embedder.encode_sparse` / :meth:`encode`
-are provided for a future sparse-retrieval layer but are not yet consumed by the
-pipeline (audit C-14).
-
-Notes for callers
------------------
-* Empty input is handled explicitly — no model call is issued.
-* Dense vectors are L2-normalised row-wise so cosine similarity reduces
-  to a dot product.
-* :func:`get_embedder` returns a process-wide singleton so the model is
-  loaded at most once per Python process.
-"""
-
-from __future__ import annotations
-
-import numpy as np
-import torch
-from FlagEmbedding import BGEM3FlagModel
-
-
-class Embedder:
-    """Thin wrapper over :class:`BGEM3FlagModel` with a stable API.
-
-    The wrapper hides the FlagEmbedding-specific encode-dict and exposes
-    three operations relevant to the ingestion pipeline:
-
-    * :meth:`encode_dense` — dense float32 matrix, L2-normalised.
-    * :meth:`encode_sparse` — list of ``{token_id: weight}`` dicts.
-    * :meth:`encode` — both in a single forward pass.
-    """
-
-    def __init__(
-        self,
-        model_name: str = "BAAI/bge-m3",
-        dim: int = 1024,
-        device: str | None = None,
-        batch_size: int = 32,
-    ) -> None:
-        resolved_device = device or ("cuda" if torch.cuda.is_available() else "cpu")
-        self._model = BGEM3FlagModel(
-            model_name,
-            use_fp16=False,
-            devices=[resolved_device],
-        )
-        self.dim = dim
-        self.batch_size = batch_size
-
-    # ------------------------------------------------------------------
-    # Internal helpers
-    # ------------------------------------------------------------------
-    @staticmethod
-    def _l2_normalise(matrix: np.ndarray) -> np.ndarray:
-        """Return ``matrix`` with every row scaled to unit L2 norm.
-
-        Zero-rows are left untouched (we divide by 1 rather than 0 to
-        avoid ``nan``s — the BGE-M3 encoder never emits zero vectors in
-        practice, but the guard keeps the function total).
-        """
-        norms = np.linalg.norm(matrix, axis=1, keepdims=True)
-        norms = np.where(norms == 0.0, 1.0, norms)
-        return (matrix / norms).astype(np.float32)
-
-    @staticmethod
-    def _coerce_sparse(weights: list[dict[int | str, float]]) -> list[dict[int, float]]:
-        """Cast token ids to ``int`` and weights to ``float``.
-
-        FlagEmbedding returns ids as strings in some versions and as
-        ints in others; we normalise to ``int`` so downstream code can
-        rely on a stable key type.
-        """
-        return [
-            {int(token_id): float(weight) for token_id, weight in row.items()} for row in weights
-        ]
-
-    # ------------------------------------------------------------------
-    # Public API
-    # ------------------------------------------------------------------
-    def encode_dense(self, texts: list[str]) -> np.ndarray:
-        """Encode ``texts`` to a dense, L2-normalised float32 matrix.
-
-        Returns a ``(0, dim)`` zero-matrix for an empty input.
-        """
-        if not texts:
-            return np.zeros((0, self.dim), dtype=np.float32)
-
-        result = self._model.encode(
-            texts,
-            batch_size=self.batch_size,
-            return_dense=True,
-            return_sparse=False,
-        )
-        dense = np.asarray(result["dense_vecs"], dtype=np.float32)
-        return self._l2_normalise(dense)
-
-    def encode_sparse(self, texts: list[str]) -> list[dict[int, float]]:
-        """Encode ``texts`` to a list of sparse ``{token_id: weight}`` dicts.
-
-        Returns ``[]`` for an empty input.
-        """
-        if not texts:
-            return []
-
-        result = self._model.encode(
-            texts,
-            batch_size=self.batch_size,
-            return_dense=False,
-            return_sparse=True,
-        )
-        return self._coerce_sparse(result["lexical_weights"])
-
-    def encode(self, texts: list[str]) -> tuple[np.ndarray, list[dict[int, float]]]:
-        """Encode ``texts`` to dense **and** sparse in one forward pass.
-
-        Returns ``(zeros((0, dim)), [])`` for an empty input.
-        """
-        if not texts:
-            return np.zeros((0, self.dim), dtype=np.float32), []
-
-        result = self._model.encode(
-            texts,
-            batch_size=self.batch_size,
-            return_dense=True,
-            return_sparse=True,
-        )
-        dense = self._l2_normalise(np.asarray(result["dense_vecs"], dtype=np.float32))
-        sparse = self._coerce_sparse(result["lexical_weights"])
-        return dense, sparse
-
-
-# ---------------------------------------------------------------------------
-# Process-wide singleton
-# ---------------------------------------------------------------------------
-_embedder: Embedder | None = None
-
-
-def get_embedder() -> Embedder:
-    """Return the process-wide :class:`Embedder` singleton.
-
-    The first call constructs the embedder using values from
-    :class:`codex.config.Settings`; subsequent calls return the cached
-    instance.  Tests can reset the cache by setting
-    ``codex.embed._embedder`` back to ``None``.
-    """
-    global _embedder
-    if _embedder is None:
-        from codex.config import Settings
-
-        s = Settings()
-        _embedder = Embedder(model_name=s.embedding_model, dim=s.embedding_dim)
-    return _embedder

codex/graph.py

@@ -1,209 +0,0 @@
-"""Citation graph analytics (F-15).
-
-Builds an in-memory NetworkX DiGraph from the ``citations`` table and
-provides PageRank, bibliographic coupling, co-citation, and dangling-
-citation queries.  The graph is ephemeral — rebuilt per call, < 1 s for
-≤ 100 papers.
-
-Graceful degradation
---------------------
-* ``citation_pagerank`` returns uniform scores when the graph has fewer
-  than 5 nodes, and logs a warning.
-* All functions accept an empty graph without raising.
-"""
-
-from __future__ import annotations
-
-import logging
-from typing import Any, cast
-
-import networkx as nx
-
-logger = logging.getLogger(__name__)
-
-_MIN_PAGERANK_NODES = 5
-
-# Single source of truth for resolving ``citations.cited_id`` to a canonical
-# paper id. ``cited_id`` stores OpenAlex IDs while ``papers.id`` stores DOIs /
-# arXiv ids, so an in-KB reference cited by its OpenAlex id does not match
-# ``papers.id`` directly. This resolves it to the canonical ``papers.id`` (via
-# ``papers.openalex_id``, then the ``paper_identifiers`` alias table); dangling
-# references keep their raw id. Both :func:`build_citation_graph` and
-# :mod:`codex.discover` query through this so the "what counts as ingested" rule
-# cannot drift between them again (audit C-1). Trusted constant — no user input.
-RESOLVED_CITATIONS_SQL = """
-    SELECT c.citing_id,
-           COALESCE(p.id, pi.paper_id, c.cited_id) AS cited_id
-    FROM citations c
-    LEFT JOIN papers p ON p.openalex_id = c.cited_id
-    LEFT JOIN paper_identifiers pi
-           ON pi.openalex_id = c.cited_id AND p.id IS NULL
-"""
-
-
-def build_citation_graph(conn: Any) -> nx.DiGraph:
-    """Load the ``citations`` table into a directed graph.
-
-    Nodes are paper IDs (strings).  An edge ``citing → cited`` means
-    the citing paper references the cited paper.  Both ingested papers
-    and dangling targets (cited but not yet ingested) appear as nodes.
-
-    ``cited_id`` in the citations table stores OpenAlex IDs; ``papers.id``
-    stores DOIs.  A LEFT JOIN resolves cited papers that are in the KB to
-    their canonical DOI key via ``papers.openalex_id``.  Dangling references
-    (not in KB) keep their raw OpenAlex ID.
-
-    Parameters
-    ----------
-    conn:
-        Open psycopg connection (dict-row factory assumed).
-
-    Returns
-    -------
-    nx.DiGraph with every (citing_id, cited_id) pair as an edge.
-    """
-    rows = conn.execute(RESOLVED_CITATIONS_SQL).fetchall()
-    g: nx.DiGraph = nx.DiGraph()
-    for row in rows:
-        g.add_edge(row["citing_id"], row["cited_id"])
-    logger.debug(
-        "Built citation graph: %d nodes, %d edges", g.number_of_nodes(), g.number_of_edges()
-    )
-    return g
-
-
-def citation_pagerank(graph: nx.DiGraph, *, damping: float = 0.85) -> dict[str, float]:
-    """Compute PageRank over the citation graph.
-
-    Parameters
-    ----------
-    graph:
-        DiGraph from :func:`build_citation_graph`.
-    damping:
-        PageRank damping factor (default 0.85).
-
-    Returns
-    -------
-    ``{paper_id: score}`` dict.  Scores sum to approximately 1.0.
-
-    When ``graph`` has fewer than :data:`_MIN_PAGERANK_NODES` (5) nodes,
-    returns a uniform distribution and logs a warning — the graph is too
-    sparse for the random-walk model to converge meaningfully.  The
-    ``damping`` parameter is ignored in this branch.  The user-facing
-    corpus-size threshold (default 15) lives in
-    :attr:`codex.config.Settings.graph_min_corpus_size` and is surfaced
-    by ``codex graph report``.
-    """
-    n = graph.number_of_nodes()
-    if n == 0:
-        return {}
-    if n < _MIN_PAGERANK_NODES:
-        logger.warning(
-            "citation_pagerank: only %d nodes — corpus too small for meaningful ranking "
-            "(need ≥ %d). Returning uniform scores.",
-            n,
-            _MIN_PAGERANK_NODES,
-        )
-        uniform = 1.0 / n
-        return {node: uniform for node in graph.nodes()}
-    return cast(dict[str, float], nx.pagerank(graph, alpha=damping))
-
-
-def citing_coverage(graph: nx.DiGraph, known_ids: set[str]) -> tuple[int, int]:
-    """Return ``(papers_with_out_edges, total_papers)`` — the citing-paper coverage.
-
-    A paper "covers" the citation graph when it has ≥1 out-edge (it cites at least
-    one work). Low citing coverage starves the F-15 layer (PageRank / coupling /
-    co-citation) even when the total paper count looks healthy, because those
-    metrics are driven by out-edges, not by node count. Surfaced by
-    ``codex graph report`` (R-D); the warning threshold is
-    :attr:`codex.config.Settings.graph_min_citing_coverage`.
-    """
-    n_citing = sum(1 for pid in known_ids if pid in graph and graph.out_degree(pid) > 0)
-    return n_citing, len(known_ids)
-
-
-def find_related(paper_id: str, graph: nx.DiGraph, *, min_shared: int = 2) -> list[str]:
-    """Bibliographic coupling: papers sharing ≥ ``min_shared`` references with ``paper_id``.
-
-    Two papers are bibliographically coupled when they cite the same sources.
-    The more shared references, the more likely they treat related topics.
-
-    Parameters
-    ----------
-    paper_id:
-        Source paper whose references form the seed set.
-    graph:
-        DiGraph from :func:`build_citation_graph`.
-    min_shared:
-        Minimum number of shared references to be included.
-
-    Returns
-    -------
-    List of paper IDs ordered by shared-reference count DESC.
-    """
-    if paper_id not in graph:
-        return []
-    # papers cited by paper_id
-    references: set[str] = set(graph.successors(paper_id))
-    if not references:
-        return []
-
-    shared: dict[str, int] = {}
-    for ref in references:
-        # other papers that also cite this reference
-        for co_citer in graph.predecessors(ref):
-            if co_citer == paper_id:
-                continue
-            shared[co_citer] = shared.get(co_citer, 0) + 1
-
-    return [
-        pid for pid, count in sorted(shared.items(), key=lambda x: -x[1]) if count >= min_shared
-    ]
-
-
-def find_co_cited(paper_id: str, graph: nx.DiGraph) -> list[tuple[str, int]]:
-    """Co-citation: papers frequently cited alongside ``paper_id``.
-
-    A paper X is co-cited with ``paper_id`` when some third paper cites both.
-
-    Returns
-    -------
-    ``[(paper_id, count)]`` ordered by count DESC.
-    """
-    if paper_id not in graph:
-        return []
-    # papers that cite paper_id
-    citers: set[str] = set(graph.predecessors(paper_id))
-    if not citers:
-        return []
-
-    co_cited: dict[str, int] = {}
-    for citer in citers:
-        for other in graph.successors(citer):
-            if other == paper_id:
-                continue
-            co_cited[other] = co_cited.get(other, 0) + 1
-
-    return sorted(co_cited.items(), key=lambda x: -x[1])
-
-
-def dangling_citations(graph: nx.DiGraph, known_ids: set[str]) -> list[str]:
-    """Return graph nodes not present in ``known_ids`` (cited but not yet ingested).
-
-    Complements :func:`codex.discover.discovery_leads` with graph context:
-    the returned IDs are reachable in the citation graph but have no paper
-    record in the DB.
-
-    Parameters
-    ----------
-    graph:
-        DiGraph from :func:`build_citation_graph`.
-    known_ids:
-        Set of paper IDs that exist in the ``papers`` table.
-
-    Returns
-    -------
-    List of dangling paper IDs (unordered).
-    """
-    return [node for node in graph.nodes() if node not in known_ids]

codex/ingest.py

@@ -1,505 +0,0 @@
-"""End-to-end idempotent ingest pipeline for a single paper."""
-
-from __future__ import annotations
-
-import logging
-from dataclasses import dataclass
-from pathlib import Path
-
-import numpy as np
-import psycopg
-
-from codex.config import get_settings
-from codex.db import get_conn
-from codex.embed import get_embedder
-from codex.models import Citation, Paper
-from codex.quality import is_quality_chunk, section_label
-from codex.sources import arxiv, crossref, openalex, semanticscholar
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass
-class IngestResult:
-    paper_id: str
-    chunks_upserted: int
-    citations_upserted: int
-    formulas_upserted: int = 0
-    figures_upserted: int = 0
-
-
-def _make_bibkey(paper: Paper) -> str | None:
-    """Generate a BibTeX-style key from paper metadata.
-
-    Format: ConcatSurnames + Year (e.g. "BobenkoPinkallSpringborn2015").
-    Surnames are the last whitespace-separated token of each author name.
-    >3 authors: FirstSurname + "EtAl" + Year.
-    Returns None when authors or year are missing.
-    """
-    if not paper.authors or not paper.year:
-        return None
-    surnames = [name.split()[-1] for name in paper.authors if name.strip()]
-    if not surnames:
-        return None
-    year = str(paper.year)
-    if len(surnames) <= 3:
-        return "".join(surnames) + year
-    return surnames[0] + "EtAl" + year
-
-
-def _s2_id_for(pid: str) -> str | None:
-    """Format a canonical paper id for the Semantic Scholar paper endpoint.
-
-    S2 needs a namespaced id (``arXiv:<id>``, ``DOI:<doi>``) or a bare 40-hex
-    S2 paperId — a bare arXiv id like ``2305.10988`` 404s. DOIs are detected by
-    the ``10.`` prefix; everything else is assumed to be an arXiv id (modern
-    ``2305.10988`` or legacy ``math/0603097``).
-    """
-    p = (pid or "").strip()
-    if not p:
-        return None
-    if p.lower().startswith(("arxiv:", "doi:")):
-        return p
-    if p.startswith("10."):
-        return f"DOI:{p}"
-    return f"arXiv:{p}"
-
-
-def _norm_cited_id(cited_id: str) -> str:
-    """Normalize a DOI cited-id to its canonical bare, lower-cased form.
-
-    DOIs are case-insensitive and may arrive bare (``10.…``), as a
-    ``https://doi.org/…`` URL, or ``doi:…``-prefixed: GROBID emits whichever
-    form the PDF reference text used, while S2/OpenAlex hand back bare DOIs.
-    Stripping the prefix and lower-casing keeps the citation-graph join from
-    fragmenting the same reference into distinct case-/URL-variant nodes. arXiv
-    ids and S2 paperIds are left untouched. Mirrors ``openalex._normalize_doi``."""
-    s = cited_id.strip()
-    lowered = s.lower()
-    for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
-        if lowered.startswith(prefix):
-            return lowered[len(prefix) :]
-    return lowered if lowered.startswith("10.") else s
-
-
-def _recover_abstract(paper: Paper) -> str | None:
-    """Recover a missing abstract from S2, then Crossref (DQ-2 supplement).
-
-    Tried in order of coverage for this corpus: Semantic Scholar (indexes most
-    arXiv + journal abstracts), then Crossref (publisher-deposited, DOI only).
-    Network/parse failures degrade to None rather than aborting the ingest.
-    """
-    s2_id = _s2_id_for(paper.id)
-    if s2_id is not None:
-        try:
-            abstract = semanticscholar.fetch_abstract(s2_id)
-            if abstract and abstract.strip():
-                return abstract.strip()
-        except Exception:
-            logger.warning("S2 abstract recovery failed for %s", paper.id, exc_info=True)
-    if paper.id.startswith("10."):
-        try:
-            abstract = crossref.fetch_abstract(paper.id)
-            if abstract and abstract.strip():
-                return abstract.strip()
-        except Exception:
-            logger.warning("Crossref abstract recovery failed for %s", paper.id, exc_info=True)
-    return None
-
-
-def _s2_reference_supplement(paper: Paper) -> list[Citation]:
-    """Fetch a paper's references from Semantic Scholar (DQ-1 supplement).
-
-    Used when OpenAlex returns an empty ``referenced_works`` list — common for
-    arXiv preprints and institutional theses that OpenAlex indexes without a
-    parsed bibliography. ``citing_id`` is rewritten to the canonical ``paper.id``
-    so the FK holds, and cited DOIs are case-normalised. Network/parse failures
-    degrade to an empty list rather than aborting the ingest.
-    """
-    s2_id = _s2_id_for(paper.id)
-    if s2_id is None:
-        return []
-    try:
-        raw = semanticscholar.fetch_references(s2_id)
-    except Exception:
-        logger.warning("S2 reference supplement failed for %s", paper.id, exc_info=True)
-        return []
-    return [
-        Citation(citing_id=paper.id, cited_id=_norm_cited_id(c.cited_id), context=c.context)
-        for c in raw
-        if c.cited_id
-    ]
-
-
-def _crossref_reference_supplement(paper: Paper) -> list[Citation]:
-    """Fetch a paper's references from Crossref (roadmap R-B supplement).
-
-    Third leg of the citation fallback chain (OpenAlex-empty → S2 → Crossref).
-    Crossref carries publisher-deposited reference lists keyed on the citing
-    work's DOI, so only DOI papers can be looked up (arXiv-only ids → []).
-    ``citing_id`` is rewritten to the canonical ``paper.id`` and cited DOIs are
-    case-normalised. Network/parse failures degrade to [] rather than aborting.
-    """
-    if not paper.id.startswith("10."):
-        return []
-    try:
-        raw = crossref.fetch_references(paper.id)
-    except Exception:
-        logger.warning("Crossref reference supplement failed for %s", paper.id, exc_info=True)
-        return []
-    return [
-        Citation(citing_id=paper.id, cited_id=_norm_cited_id(c.cited_id), context=c.context)
-        for c in raw
-        if c.cited_id
-    ]
-
-
-def ingest_paper(
-    paper_id: str,
-    source_path: str | None = None,
-    rich: bool = False,
-) -> IngestResult:
-    """Idempotent ingest of one paper.
-
-    Parameters
-    ----------
-    paper_id:
-        An arXiv ID (``"2301.07041"``), DOI (``"10.1145/…"``), or
-        OpenAlex W-ID (``"W2741809807"``).
-    source_path:
-        Optional path to a local ``.tex`` or ``.pdf`` file.
-        If given, the file is parsed into text chunks and stored.
-        Supports:
-        - ``.tex``  → :func:`codex.parsing.tex.latex_to_text` + chunk
-        - ``.pdf``  → :func:`codex.parsing.nougat.pdf_to_markdown` + chunk
-                      + :func:`codex.parsing.grobid.extract_references` for refs
-    rich:
-        When True and *source_path* is a PDF, also extract formulas via
-        :func:`codex.parsing.mathpix.extract_formulas` and figures via
-        :func:`codex.parsing.figures.extract_figures` (F-09).
-
-    Returns
-    -------
-    IngestResult
-        Counts of upserted chunks, citations, formulas, and figures.
-    """
-    # ---------------------------------------------------------------
-    # 1. Fetch metadata (OpenAlex primary, SemanticScholar fallback)
-    # ---------------------------------------------------------------
-    paper: Paper | None = openalex.fetch_paper(paper_id)
-
-    if paper is None:
-        # Detect arXiv IDs: numeric pattern like "2301.07041" or "arxiv:..." prefix
-        pid_lower = paper_id.lower()
-        looks_like_arxiv = pid_lower.startswith("arxiv:") or (
-            len(paper_id) > 4 and paper_id[4:5] == "." and paper_id[:4].isdigit()
-        )
-        if looks_like_arxiv:
-            # OpenAlex 404 on an arXiv id → recover authoritative metadata from
-            # the arXiv API (DQ-2); fall back to a minimal stub if arXiv also
-            # has nothing. Citation recovery from S2 happens in the citation
-            # block below (DQ-1 supplement).
-            paper = arxiv.fetch_metadata(paper_id) or Paper(id=paper_id, title="")
-        else:
-            raise ValueError(f"Paper not found: {paper_id}")
-
-    if paper is None:
-        raise ValueError(f"Paper not found: {paper_id}")
-
-    # Canonical identity is the caller-supplied id — what the CLI / ingest scripts
-    # use and what papers.id is contracted to hold (arXiv id or DOI). Pin it to the
-    # caller's id (audit C-7), but normalize it the same way cited-ids are (strip a
-    # https://doi.org/ or doi: prefix, lower-case DOIs) so a non-bare caller cannot
-    # store a URL-form papers.id that would defeat DQ-5's ON CONFLICT (id) idempotency
-    # or the `paper.id.startswith("10.")` recovery gates below. arXiv/W ids pass
-    # through unchanged. The OpenAlex work id is retained as openalex_id / in
-    # paper_identifiers.
-    paper.id = _norm_cited_id(paper_id.strip())
-
-    # DQ-2: OpenAlex often has the paper but no abstract (it cannot redistribute
-    # some publishers' abstracts). Supplement from S2, then Crossref (uses the
-    # canonical paper.id set above), so the paper gets a real (non-zero) abstract
-    # embedding instead of a zero vector.
-    if not (paper.abstract or "").strip():
-        recovered = _recover_abstract(paper)
-        if recovered:
-            paper.abstract = recovered
-
-    # Auto-generate bibkey when source has none (D-04).
-    if paper.bibkey is None:
-        paper.bibkey = _make_bibkey(paper)
-
-    # ---------------------------------------------------------------
-    # 2. Embed abstract (dense only — schema has one vector column)
-    # ---------------------------------------------------------------
-    embedder = get_embedder()
-    abstract_text = paper.abstract or ""
-    if abstract_text:
-        abstract_emb_arr: np.ndarray = embedder.encode_dense([abstract_text])
-        abstract_emb: list[float] = abstract_emb_arr[0].tolist()
-    else:
-        dim = embedder.dim
-        abstract_emb = [0.0] * dim
-
-    # ---------------------------------------------------------------
-    # 3. Upsert paper  (INSERT … ON CONFLICT (id) DO UPDATE SET …)
-    # ---------------------------------------------------------------
-    upsert_sql = """
-        INSERT INTO papers (id, openalex_id, bibkey, title, authors, year, abstract,
-                            source_path, abstract_emb)
-        VALUES (%(id)s, %(openalex_id)s, %(bibkey)s, %(title)s, %(authors)s, %(year)s,
-                %(abstract)s, %(source_path)s, %(abstract_emb)s)
-        ON CONFLICT (id) DO UPDATE SET
-            openalex_id  = EXCLUDED.openalex_id,
-            bibkey       = COALESCE(papers.bibkey, EXCLUDED.bibkey),
-            title        = EXCLUDED.title,
-            authors      = EXCLUDED.authors,
-            year         = EXCLUDED.year,
-            abstract     = EXCLUDED.abstract,
-            source_path  = COALESCE(EXCLUDED.source_path, papers.source_path),
-            abstract_emb = EXCLUDED.abstract_emb
-    """
-    with get_conn() as conn:
-        # The id-keyed upsert does not catch a papers.bibkey UNIQUE collision (two
-        # same-author/year papers generate the same key). On such a collision retry
-        # with an a/b/c… suffix instead of failing the whole ingest (audit C-15).
-        base_bibkey = paper.bibkey
-        for attempt in range(27):
-            try:
-                conn.execute(
-                    upsert_sql,
-                    {
-                        "id": paper.id,
-                        "openalex_id": paper.openalex_id,
-                        "bibkey": paper.bibkey,
-                        "title": paper.title,
-                        "authors": paper.authors,
-                        "year": paper.year,
-                        "abstract": paper.abstract,
-                        "source_path": source_path,
-                        "abstract_emb": abstract_emb,
-                    },
-                )
-                break
-            except psycopg.errors.UniqueViolation as exc:
-                msg = str(exc).lower()
-                conn.rollback()
-                if "openalex" in msg:
-                    # Two different papers.id claim the same OpenAlex work — a real data
-                    # conflict, not something to auto-rename. Surface it clearly instead
-                    # of the raw psycopg error (companion gap to C-15's bibkey case).
-                    raise ValueError(
-                        f"openalex_id {paper.openalex_id!r} is already attached to a "
-                        f"different paper; ingesting '{paper.id}' would duplicate it — "
-                        "resolve the conflicting paper first."
-                    ) from exc
-                if base_bibkey is None or "bibkey" not in msg or attempt == 26:
-                    raise
-                paper.bibkey = f"{base_bibkey}{chr(ord('a') + attempt)}"
-                logger.warning(
-                    "bibkey %r already exists; ingesting '%s' as %r instead",
-                    base_bibkey,
-                    paper.id,
-                    paper.bibkey,
-                )
-
-        # Register openalex_id in paper_identifiers (alias table).
-        # Every ingest records the primary openalex_id so the graph JOIN
-        # resolves it even if papers.openalex_id is later updated.
-        if paper.openalex_id:
-            conn.execute(
-                """
-                INSERT INTO paper_identifiers (paper_id, openalex_id)
-                VALUES (%s, %s)
-                ON CONFLICT DO NOTHING
-                """,
-                (paper.id, paper.openalex_id),
-            )
-
-        # ---------------------------------------------------------------
-        # 4. Parse + chunk + embed source file (if provided)
-        #    DELETE existing chunks for this paper first, then bulk INSERT
-        # ---------------------------------------------------------------
-        chunks_upserted = 0
-        pdf_citations: list[Citation] = []
-
-        if source_path is not None:
-            from codex.parsing.grobid import extract_references
-            from codex.parsing.nougat import pdf_to_markdown
-            from codex.parsing.tex import chunk_sections, chunk_text
-
-            suffix = Path(source_path).suffix.lower()
-            # (section_title | None, chunk_text) pairs. `.tex` is section-aware so
-            # the stored `section` column reflects the real heading (R-C / R-12);
-            # `.txt`/`.pdf` carry no section structure, so the title is None.
-            raw_pairs: list[tuple[str | None, str]] = []
-
-            if suffix == ".tex":
-                raw_pairs = chunk_sections(Path(source_path).read_text())
-            elif suffix == ".txt":
-                text = Path(source_path).read_text(encoding="utf-8", errors="replace")
-                raw_pairs = [(None, c) for c in chunk_text(text)]
-            elif suffix == ".pdf":
-                raw_pairs = [(None, c) for c in chunk_text(pdf_to_markdown(source_path))]
-                # Also extract GROBID refs
-                grobid_refs = extract_references(source_path)
-                for ref in grobid_refs:
-                    cited_id = ref.get("doi") or ref.get("arxiv_id")
-                    if cited_id:
-                        # Normalize the GROBID DOI (bare/URL/``doi:`` → bare lower-case)
-                        # so PDF-derived edges join the same graph nodes as the
-                        # S2/OpenAlex paths; arXiv ids pass through untouched.
-                        pdf_citations.append(
-                            Citation(citing_id=paper.id, cited_id=_norm_cited_id(cited_id))
-                        )
-            else:
-                logger.warning("Unbekannter Dateityp: %s — kein Text-Parsing", source_path)
-
-            # Quality-gate the pairs (keeps each chunk aligned with its section title).
-            settings = get_settings()
-            kept_pairs = [
-                (title, content)
-                for title, content in raw_pairs
-                if is_quality_chunk(content, settings=settings)
-            ]
-
-            # Delete existing chunks for this paper
-            conn.execute("DELETE FROM chunks WHERE paper_id = %s", (paper.id,))
-
-            if kept_pairs:
-                # Embed all chunks in one batch
-                chunk_contents = [content for _, content in kept_pairs]
-                chunk_embeddings = embedder.encode_dense(chunk_contents)
-                chunk_rows = [
-                    (
-                        paper.id,
-                        ord_idx,
-                        content,
-                        chunk_embeddings[ord_idx].tolist(),
-                        # .tex: label from the real \section heading — keeps the
-                        # controlled bucket when it matches, else stores the cleaned
-                        # title (R-F); .txt/.pdf (title=None) classify by content.
-                        # NB: `section` semantics now differ by source (write-only col).
-                        section_label(title, content),
-                    )
-                    for ord_idx, (title, content) in enumerate(kept_pairs)
-                ]
-                with conn.cursor() as cur:
-                    cur.executemany(
-                        "INSERT INTO chunks (paper_id, ord, content, embedding, section)"
-                        " VALUES (%s, %s, %s, %s, %s)",
-                        chunk_rows,
-                    )
-                chunks_upserted = len(chunk_rows)
-
-        # ---------------------------------------------------------------
-        # 5. Fetch citations  (fallback chain: OpenAlex → S2 → Crossref)
-        #    Insert with ON CONFLICT (citing_id, cited_id) DO NOTHING
-        # ---------------------------------------------------------------
-        api_citations: list[Citation]
-        if paper.openalex_id:
-            raw = openalex.fetch_citations(paper.openalex_id)
-            # OpenAlex returns citing_id=openalex_id, but papers.id uses the DOI.
-            # Rewrite citing_id to paper.id so the FK constraint is satisfied.
-            api_citations = [
-                Citation(citing_id=paper.id, cited_id=c.cited_id, context=c.context) for c in raw
-            ]
-            # DQ-1: OpenAlex indexes many arXiv preprints / theses WITHOUT a
-            # parsed reference list (referenced_works == []). Fall back to the
-            # Semantic Scholar references, which often has them.
-            if not api_citations:
-                api_citations = _s2_reference_supplement(paper)
-        else:
-            api_citations = _s2_reference_supplement(paper)
-        # R-B: if OpenAlex and S2 both came back empty, try Crossref's
-        # publisher-deposited references (DOI papers only) as a third leg.
-        if not api_citations:
-            api_citations = _crossref_reference_supplement(paper)
-
-        # Merge API citations and GROBID PDF citations; dedup via set
-        all_citations_set: set[tuple[str, str]] = set()
-        merged_citations: list[Citation] = []
-        for cit in api_citations + pdf_citations:
-            key = (cit.citing_id, cit.cited_id)
-            if key not in all_citations_set:
-                all_citations_set.add(key)
-                merged_citations.append(cit)
-
-        if merged_citations:
-            with conn.cursor() as cur:
-                cur.executemany(
-                    """
-                    INSERT INTO citations (citing_id, cited_id, context)
-                    VALUES (%s, %s, %s)
-                    ON CONFLICT (citing_id, cited_id) DO NOTHING
-                    """,
-                    [(c.citing_id, c.cited_id, c.context) for c in merged_citations],
-                )
-
-        citations_upserted = len(merged_citations)
-        conn.commit()
-
-    # ---------------------------------------------------------------
-    # 6. F-09 Rich Parsing: formulas + figures (PDF only)
-    # ---------------------------------------------------------------
-    formulas_upserted = 0
-    figures_upserted = 0
-
-    if rich and source_path is not None and Path(source_path).suffix.lower() == ".pdf":
-        from codex.parsing.figures import extract_figures
-        from codex.parsing.mathpix import extract_formulas
-
-        settings = get_settings()
-
-        formulas = extract_formulas(source_path)
-        figures = extract_figures(source_path, output_dir=settings.figures_dir)
-
-        if formulas or figures:
-            with get_conn() as conn2:
-                with conn2.cursor() as cur:
-                    # Delete-before-insert keeps re-ingest idempotent (BIGSERIAL has no
-                    # natural UNIQUE key, so ON CONFLICT DO NOTHING never fires).
-                    cur.execute("DELETE FROM formulas WHERE paper_id = %s", (paper.id,))
-                    cur.execute("DELETE FROM figures  WHERE paper_id = %s", (paper.id,))
-
-                if formulas:
-                    with conn2.cursor() as cur:
-                        cur.executemany(
-                            """
-                            INSERT INTO formulas (paper_id, page, raw_latex, context, eq_label)
-                            VALUES (%s, %s, %s, %s, %s)
-                            """,
-                            [
-                                # paper.id, not f.paper_id: the extractor derives its
-                                # paper_id from the PDF filename stem, which need not match
-                                # papers.id and would violate the FK (audit C-10).
-                                (paper.id, f.page, f.raw_latex, f.context, f.eq_label)
-                                for f in formulas
-                            ],
-                        )
-                    formulas_upserted = len(formulas)
-
-                if figures:
-                    with conn2.cursor() as cur:
-                        cur.executemany(
-                            """
-                            INSERT INTO figures (paper_id, page, caption, image_path)
-                            VALUES (%s, %s, %s, %s)
-                            """,
-                            [
-                                # paper.id, not fig.paper_id (filename stem) — see C-10 above.
-                                (paper.id, fig.page, fig.caption, fig.image_path)
-                                for fig in figures
-                            ],
-                        )
-                    figures_upserted = len(figures)
-
-                conn2.commit()
-
-    return IngestResult(
-        paper_id=paper.id,
-        chunks_upserted=chunks_upserted,
-        citations_upserted=citations_upserted,
-        formulas_upserted=formulas_upserted,
-        figures_upserted=figures_upserted,
-    )

codex/mcp_server.py

@@ -1,358 +0,0 @@
-"""MCP server — read-only KB tools exposed via FastMCP (F-14).
-
-This module is a *thin facade* over existing codex domain modules.
-No new domain logic lives here — each tool delegates to the relevant
-module and reshapes the result into a JSON-serialisable dict/list.
-
-Graceful degradation contract
-------------------------------
-Every tool that depends on an optional feature (wiki, leads/synthesis,
-provenance-verify) catches all exceptions and returns
-``{"error": "feature not available"}`` instead of crashing.
-
-Transport
----------
-* **stdio** (default):
-  ``python -m codex.mcp_server``  or  ``codex-mcp``
-
-* **HTTP** (config-gated):
-  Set ``MCP_TRANSPORT=http`` *and* ``MCP_AUTH_TOKEN=<token>`` in your
-  environment (or ``.env``).  The server refuses to start without a
-  token — no unauthenticated HTTP endpoints are ever exposed.
-"""
-
-from __future__ import annotations
-
-import logging
-import secrets
-
-from mcp.server.fastmcp import FastMCP
-from mcp.server.transport_security import TransportSecuritySettings
-
-logger = logging.getLogger(__name__)
-
-# DNS rebinding protection disabled: the server is guarded by Bearer token auth
-# (mcp_server.py _TokenAuth middleware) and UFW firewall (trusted LAN IPs only).
-mcp = FastMCP(
-    "codex",
-    transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False),
-)
-
-
-# ---------------------------------------------------------------------------
-# Tool: search
-# ---------------------------------------------------------------------------
-
-
-@mcp.tool()
-def search(query: str, limit: int = 10) -> list[dict[str, object]]:
-    """Chunk-level hybrid search (dense BGE-M3 + FTS) over the KB.
-
-    Returns one dict per hit:
-    ``{bibkey, paper_id, locator, score, snippet}``.
-    """
-    try:
-        from codex.db import get_conn
-        from codex.embed import get_embedder
-
-        embedder = get_embedder()
-        dense_vec = embedder.encode_dense([query])[0].tolist()
-
-        sql = """
-            SELECT
-                c.id,
-                c.paper_id,
-                c.ord,
-                c.content,
-                p.bibkey,
-                c.embedding <-> %(emb)s::vector AS dist
-            FROM chunks c
-            JOIN papers p ON p.id = c.paper_id
-            WHERE c.embedding IS NOT NULL
-              AND p.bibkey IS NOT NULL
-            ORDER BY c.embedding <-> %(emb)s::vector
-            LIMIT %(limit)s
-        """
-        with get_conn() as conn:
-            rows = conn.execute(sql, {"emb": dense_vec, "limit": limit}).fetchall()
-
-        results: list[dict[str, object]] = []
-        for row in rows:
-            results.append(
-                {
-                    "bibkey": row["bibkey"],
-                    "paper_id": row["paper_id"],
-                    "locator": f"chunk {row['ord']}",
-                    "score": float(1.0 - row["dist"]),  # convert distance → similarity
-                    "snippet": row["content"][:300],
-                }
-            )
-        return results
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("search tool error: %s", exc)
-        return [{"error": str(exc)}]
-
-
-# ---------------------------------------------------------------------------
-# Tool: ask
-# ---------------------------------------------------------------------------
-
-
-@mcp.tool()
-def ask(question: str) -> dict[str, object]:
-    """RAG answer over the KB.
-
-    Returns ``{"answer": str}`` when implemented,
-    or ``{"error": "not available"}`` until a RAG layer is added.
-    """
-    # RAG not yet implemented (CLI parity with codex ask)
-    return {"error": "not available"}
-
-
-# ---------------------------------------------------------------------------
-# Tool: wiki_read
-# ---------------------------------------------------------------------------
-
-
-@mcp.tool()
-def wiki_read(concept_slug: str) -> dict[str, object]:
-    """Compiled concept page (F-12) as ``{markdown, sources}``.
-
-    Gracefully returns ``{"error": "feature not available"}`` when the wiki
-    directory or the requested page does not exist.
-    """
-    try:
-        from pathlib import Path
-
-        from codex.config import get_settings
-
-        settings = get_settings()
-        wiki_dir = Path(settings.wiki_dir)
-        page_path = wiki_dir / f"{concept_slug}.md"
-
-        if not page_path.exists():
-            return {"error": "feature not available"}
-
-        markdown = page_path.read_text(encoding="utf-8")
-
-        # Collect the actual cited bibkeys from the page's inline [BibKey #loc]
-        # citations (audit C-3: this previously returned the concept slug itself,
-        # not the cited sources).
-        from codex.wiki import _parse_claims
-
-        sources = sorted({claim.bibkey for claim in _parse_claims(markdown)})
-
-        return {"markdown": markdown, "sources": sources}
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("wiki_read tool error: %s", exc)
-        return {"error": "feature not available"}
-
-
-# ---------------------------------------------------------------------------
-# Tool: wiki_list
-# ---------------------------------------------------------------------------
-
-
-@mcp.tool()
-def wiki_list() -> list[dict[str, object]]:
-    """All compiled concept pages with freshness metadata.
-
-    Returns a list of ``{slug, mtime, hash_prefix}`` dicts.
-    Gracefully returns ``[]`` when the wiki directory does not exist.
-    """
-    try:
-        import json
-        from datetime import UTC, datetime
-        from pathlib import Path
-
-        from codex.config import get_settings
-
-        settings = get_settings()
-        wiki_dir = Path(settings.wiki_dir)
-
-        if not wiki_dir.exists():
-            return []
-
-        state_path = wiki_dir / ".compile-state.json"
-        state: dict[str, str] = {}
-        if state_path.exists():
-            try:
-                state = json.loads(state_path.read_text(encoding="utf-8"))
-            except (json.JSONDecodeError, OSError):
-                state = {}
-
-        pages = sorted(wiki_dir.glob("*.md"))
-        result: list[dict[str, object]] = []
-        for page in pages:
-            if page.name in ("index.md", "log.md"):
-                continue
-            slug = page.stem
-            hash_prefix = state.get(slug, "")[:8]
-            mtime = datetime.fromtimestamp(page.stat().st_mtime, tz=UTC).isoformat()
-            result.append({"slug": slug, "mtime": mtime, "hash_prefix": hash_prefix})
-
-        return result
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("wiki_list tool error: %s", exc)
-        return []
-
-
-# ---------------------------------------------------------------------------
-# Tool: discover_leads
-# ---------------------------------------------------------------------------
-
-
-@mcp.tool()
-def discover_leads(limit: int = 20) -> list[dict[str, object]]:
-    """Dangling-citation discovery leads — papers cited but not yet ingested.
-
-    Returns ``[{cited_id, pull}]`` ordered by citation count descending.
-    """
-    try:
-        from codex.discover import discovery_leads as _leads
-
-        raw = _leads(limit=limit)
-        return [{"cited_id": row["cited_id"], "pull": int(row["pull"])} for row in raw]
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("discover_leads tool error: %s", exc)
-        return [{"error": str(exc)}]
-
-
-# ---------------------------------------------------------------------------
-# Tool: provenance_verify
-# ---------------------------------------------------------------------------
-
-
-@mcp.tool()
-def provenance_verify(lib_path: str) -> list[dict[str, object]]:
-    """Scan C++ sources for @cite tags and return code-to-paper links.
-
-    Read-only: only scans, never writes.
-    Returns ``[{file, line, symbol, bibkey}]`` or ``[{"error": ...}]``.
-    """
-    try:
-        from codex.provenance import scan_cite_tags
-
-        hits = scan_cite_tags(lib_path)
-        return [dict(h) for h in hits]
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("provenance_verify tool error: %s", exc)
-        return [{"error": "feature not available"}]
-
-
-# ---------------------------------------------------------------------------
-# Tool: synthesis_browse
-# ---------------------------------------------------------------------------
-
-
-@mcp.tool()
-def synthesis_browse(kind: str = "all") -> list[dict[str, object]]:
-    """Leads / conjectures from the ``leads/`` directory (F-13).
-
-    Reads ``.json`` files from ``leads/`` and filters by ``kind``
-    (``"all"`` returns everything).  Gracefully returns
-    ``[{"error": "feature not available"}]`` when the directory is absent.
-    """
-    try:
-        import json
-        from pathlib import Path
-
-        from codex.config import get_settings
-
-        settings = get_settings()
-        leads_dir = Path(settings.leads_dir)
-        if not leads_dir.exists():
-            return [{"error": "feature not available"}]
-
-        items: list[dict[str, object]] = []
-        for f in sorted(leads_dir.rglob("*.json")):
-            try:
-                data: dict[str, object] = json.loads(f.read_text(encoding="utf-8"))
-                item_kind = str(data.get("kind", "unknown"))
-                if kind == "all" or item_kind == kind:
-                    items.append(data)
-            except (json.JSONDecodeError, OSError):
-                continue
-
-        return items
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("synthesis_browse tool error: %s", exc)
-        return [{"error": "feature not available"}]
-
-
-# ---------------------------------------------------------------------------
-# Server entry-point helpers
-# ---------------------------------------------------------------------------
-
-
-def _require_http_token() -> str:
-    """Return the configured auth token or raise :exc:`RuntimeError`.
-
-    Called before starting HTTP transport to enforce the security invariant:
-    no unauthenticated HTTP endpoints are ever exposed.
-    """
-    from codex.config import get_settings
-
-    settings = get_settings()
-    token = settings.mcp_auth_token.get_secret_value() if settings.mcp_auth_token else ""
-    if not token:
-        raise RuntimeError(
-            "HTTP transport requires MCP_AUTH_TOKEN to be set. "
-            "Set the environment variable or add it to .env."
-        )
-    return token
-
-
-# ---------------------------------------------------------------------------
-# Server entry-point
-# ---------------------------------------------------------------------------
-
-
-def main() -> None:
-    """Start the MCP server (stdio or HTTP depending on config).
-
-    HTTP transport requires ``MCP_AUTH_TOKEN`` to be set; the server will
-    raise :exc:`RuntimeError` and refuse to start if the token is missing.
-    """
-    from codex.config import get_settings
-
-    settings = get_settings()
-    transport = settings.mcp_transport.lower()
-
-    if transport == "http":
-        import uvicorn
-        from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
-        from starlette.requests import Request
-        from starlette.responses import Response
-
-        token = _require_http_token()
-
-        class _TokenAuth(BaseHTTPMiddleware):
-            """Reject requests that do not carry a valid Bearer token."""
-
-            async def dispatch(
-                self, request: Request, call_next: RequestResponseEndpoint
-            ) -> Response:
-                auth = request.headers.get("Authorization", "")
-                # Constant-time comparison to avoid a token-length/equality timing
-                # side-channel (audit S-2).
-                if not secrets.compare_digest(auth, f"Bearer {token}"):
-                    return Response("Unauthorized", status_code=401)
-                return await call_next(request)
-
-        base_app = mcp.streamable_http_app()
-        base_app.add_middleware(_TokenAuth)
-
-        uvicorn.run(
-            base_app,
-            host=settings.mcp_host,
-            port=settings.mcp_port,
-            log_level="info",
-        )
-    else:
-        # Default: stdio transport
-        mcp.run(transport="stdio")
-
-
-if __name__ == "__main__":
-    main()

codex/models.py

@@ -78,39 +78,3 @@ class CodeLink:
     note: str | None = None
     id: int | None = None
     added_at: datetime | None = None
-
-
-@dataclass
-class FormulaChunk:
-    """Maps to the ``formulas`` table (F-09 Rich Parsing).
-
-    Stores a single extracted mathematical formula (LaTeX) from a PDF page.
-    ``id`` is set by the database (BIGSERIAL).
-    ``embedding`` is reserved for future pgvector similarity search.
-    """
-
-    paper_id: str
-    page: int
-    raw_latex: str
-    context: str
-    id: int | None = None
-    eq_label: str | None = None
-    embedding: list[float] | None = None
-
-
-@dataclass
-class FigureChunk:
-    """Maps to the ``figures`` table (F-09 Rich Parsing).
-
-    Stores metadata for a single extracted figure from a PDF page.
-    ``image_path`` points to the saved PNG on disk.
-    ``id`` is set by the database (BIGSERIAL).
-    ``embedding`` is reserved for future pgvector similarity search.
-    """
-
-    paper_id: str
-    page: int
-    image_path: str
-    caption: str
-    id: int | None = None
-    embedding: list[float] | None = None

codex/parsing/figures.py

@@ -1,169 +0,0 @@
-"""Figure extraction from PDF pages.
-
-Uses PyMuPDF (``fitz``) to iterate over embedded images in each page,
-render them to PNG and detect nearby captions using text-block heuristics.
-
-The public entry point is :func:`extract_figures`.
-"""
-
-from __future__ import annotations
-
-import logging
-from pathlib import Path
-from typing import Any
-
-from codex.models import FigureChunk
-
-logger = logging.getLogger(__name__)
-
-# Caption prefixes (case-insensitive check against text block content).
-_CAPTION_PREFIXES: tuple[str, ...] = ("figure", "fig.", "abbildung")
-
-# Maximum vertical distance (in points) from the image bottom or top
-# to consider a text block as the caption.
-_CAPTION_TOLERANCE_Y = 60.0
-
-
-# ---------------------------------------------------------------------------
-# Internal helpers
-# ---------------------------------------------------------------------------
-
-
-def _looks_like_caption(text: str) -> bool:
-    """Return True if *text* starts with a known caption prefix."""
-    lowered = text.strip().lower()
-    return any(lowered.startswith(prefix) for prefix in _CAPTION_PREFIXES)
-
-
-def _find_caption(
-    img_rect: Any,  # fitz.Rect
-    text_blocks: list[Any],
-) -> str:
-    """Find the closest caption text block to *img_rect*.
-
-    Searches text blocks that:
-    1. Start with a caption prefix (e.g. "Figure", "Fig.", "Abbildung").
-    2. Are within ``_CAPTION_TOLERANCE_Y`` points above or below the image.
-
-    Returns the first matching text, stripped; empty string if none found.
-    """
-    best: str = ""
-    best_dist = float("inf")
-
-    for block in text_blocks:
-        if len(block) < 5:
-            continue
-        bx0, by0, bx1, by1 = block[0], block[1], block[2], block[3]
-        text: str = block[4]
-
-        if not _looks_like_caption(text):
-            continue
-
-        # Distance: gap below image or gap above image
-        gap_below = by0 - img_rect.y1
-        gap_above = img_rect.y0 - by1
-
-        if 0 <= gap_below <= _CAPTION_TOLERANCE_Y:
-            dist = gap_below
-        elif 0 <= gap_above <= _CAPTION_TOLERANCE_Y:
-            dist = gap_above
-        else:
-            continue
-
-        # Horizontal overlap sanity check (caption should overlap with image)
-        overlap = min(bx1, img_rect.x1) - max(bx0, img_rect.x0)
-        if overlap < 0:
-            continue
-
-        if dist < best_dist:
-            best_dist = dist
-            best = text.replace("\n", " ").strip()
-
-    return best
-
-
-# ---------------------------------------------------------------------------
-# Public entry point
-# ---------------------------------------------------------------------------
-
-
-def extract_figures(
-    pdf_path: str,
-    output_dir: str,
-) -> list[FigureChunk]:
-    """Extract embedded figures from *pdf_path* and save them as PNG files.
-
-    For each image on each page:
-    1. Extract the image via ``page.get_images(full=True)``.
-    2. Render the image as a PNG via ``fitz.Pixmap``.
-    3. Search nearby text blocks for a caption (``Figure …`` / ``Fig. …`` /
-       ``Abbildung …``).
-    4. Save the PNG to ``<output_dir>/<paper_id>_p<page>_fig<n>.png``.
-    5. Yield a :class:`~codex.models.FigureChunk`.
-
-    Parameters
-    ----------
-    pdf_path:
-        Path to the source PDF.
-    output_dir:
-        Directory where PNG images are written (created if absent).
-
-    Returns
-    -------
-    list[FigureChunk]
-        Extracted figure metadata; empty list if no images found.
-    """
-    import fitz
-
-    results: list[FigureChunk] = []
-    paper_id = Path(pdf_path).stem
-    out_dir = Path(output_dir)
-    out_dir.mkdir(parents=True, exist_ok=True)
-
-    try:
-        doc: Any = fitz.open(pdf_path)
-    except Exception:
-        logger.exception("fitz.open failed for %s", pdf_path)
-        return results
-
-    try:
-        for page_num in range(len(doc)):
-            page = doc[page_num]
-            images: list[Any] = page.get_images(full=True)
-            text_blocks: list[Any] = page.get_text("blocks")
-
-            for fig_idx, img_info in enumerate(images):
-                xref: int = img_info[0]
-
-                # Get image bounding box on the page
-                img_rects: list[Any] = page.get_image_rects(xref)
-                if not img_rects:
-                    continue
-                img_rect = img_rects[0]
-
-                # Extract and save the image pixmap
-                try:
-                    pix: Any = fitz.Pixmap(doc, xref)
-                    if pix.n > 4:  # CMYK or similar — convert to RGB
-                        pix = fitz.Pixmap(fitz.csRGB, pix)
-                    filename = f"{paper_id}_p{page_num + 1}_fig{fig_idx + 1}.png"
-                    img_path = out_dir / filename
-                    pix.save(str(img_path))
-                except Exception:
-                    logger.debug("Could not extract image xref=%d on page %d", xref, page_num + 1)
-                    continue
-
-                caption = _find_caption(img_rect, text_blocks)
-
-                results.append(
-                    FigureChunk(
-                        paper_id=paper_id,
-                        page=page_num + 1,
-                        image_path=str(img_path),
-                        caption=caption,
-                    )
-                )
-    finally:
-        doc.close()
-
-    return results

codex/parsing/grobid.py

@@ -1,138 +0,0 @@
-"""GROBID integration.
-
-Extracts structured reference lists and full-text TEI XML from PDFs by
-calling a self-hosted GROBID HTTP server.  No DB access, no embedding,
-no network fetching of papers happens here.
-"""
-
-from __future__ import annotations
-
-import xml.etree.ElementTree as ET
-
-import httpx
-
-from codex.config import Settings
-
-_TEI_NS = "{http://www.tei-c.org/ns/1.0}"
-
-
-def _text(element: ET.Element | None) -> str:
-    """Return element text or empty string if element is None."""
-    if element is None:
-        return ""
-    return (element.text or "").strip()
-
-
-def extract_references(
-    pdf_path: str,
-    grobid_url: str | None = None,
-    timeout: float = 60.0,
-) -> list[dict[str, str]]:
-    """Extract a structured reference list from a PDF via GROBID.
-
-    Parameters
-    ----------
-    pdf_path:
-        Path to the PDF file on disk.
-    grobid_url:
-        Base URL of the GROBID server.  Defaults to ``Settings().grobid_url``.
-    timeout:
-        HTTP client timeout in seconds.  Large PDFs (e.g. theses) can exceed the
-        60s default, especially against a slower/emulated GROBID; raise it then.
-
-    Returns
-    -------
-    list[dict[str, str]]
-        One dict per reference.  All dicts contain the keys
-        ``title``, ``authors``, ``year``, ``doi``, ``arxiv_id``
-        (missing values are empty strings).
-    """
-    if grobid_url is None:
-        grobid_url = Settings().grobid_url
-
-    with open(pdf_path, "rb") as fh, httpx.Client(timeout=timeout) as client:
-        response = client.post(
-            f"{grobid_url}/api/processReferences",
-            files={"input": (pdf_path, fh, "application/pdf")},
-        )
-    response.raise_for_status()
-
-    root = ET.fromstring(response.text)
-    results: list[dict[str, str]] = []
-
-    for bib in root.iter(f"{_TEI_NS}biblStruct"):
-        # Title
-        title_el = bib.find(f".//{_TEI_NS}title[@level='a']")
-        title = _text(title_el)
-
-        # Authors: collect all persName elements
-        authors_parts: list[str] = []
-        for person in bib.iter(f"{_TEI_NS}persName"):
-            forename_el = person.find(f"{_TEI_NS}forename")
-            surname_el = person.find(f"{_TEI_NS}surname")
-            forename = _text(forename_el)
-            surname = _text(surname_el)
-            full = " ".join(p for p in (forename, surname) if p)
-            if full:
-                authors_parts.append(full)
-        authors = "; ".join(authors_parts)
-
-        # Year
-        date_el = bib.find(f".//{_TEI_NS}date[@type='published']")
-        year = ""
-        if date_el is not None:
-            when = date_el.get("when", "")
-            year = when[:4] if when else ""
-
-        # DOI
-        doi_el = bib.find(f".//{_TEI_NS}idno[@type='DOI']")
-        doi = _text(doi_el)
-
-        # arXiv ID — GROBID emits type="arXiv" (camel-case); match case-insensitively
-        arxiv_id = ""
-        for idno_el in bib.iter(f"{_TEI_NS}idno"):
-            if idno_el.get("type", "").lower() == "arxiv":
-                arxiv_id = (idno_el.text or "").strip()
-                break
-
-        results.append(
-            {
-                "title": title,
-                "authors": authors,
-                "year": year,
-                "doi": doi,
-                "arxiv_id": arxiv_id,
-            }
-        )
-
-    return results
-
-
-def extract_structure(
-    pdf_path: str,
-    grobid_url: str | None = None,
-) -> str:
-    """Extract full-text TEI XML from a PDF via GROBID.
-
-    Parameters
-    ----------
-    pdf_path:
-        Path to the PDF file on disk.
-    grobid_url:
-        Base URL of the GROBID server.  Defaults to ``Settings().grobid_url``.
-
-    Returns
-    -------
-    str
-        Raw TEI XML response text from GROBID.
-    """
-    if grobid_url is None:
-        grobid_url = Settings().grobid_url
-
-    with open(pdf_path, "rb") as fh, httpx.Client(timeout=120.0) as client:
-        response = client.post(
-            f"{grobid_url}/api/processFulltextDocument",
-            files={"input": (pdf_path, fh, "application/pdf")},
-        )
-    response.raise_for_status()
-    return response.text

codex/parsing/mathpix.py

@@ -1,319 +0,0 @@
-"""Formula extraction from PDF pages.
-
-Two backends are supported:
-
-1. **MathPix** (cloud): POST the PDF to ``api.mathpix.com/v3/pdf`` and poll
-   for the result JSON.  Requires ``MATHPIX_APP_ID`` and ``MATHPIX_APP_KEY``
-   environment variables (or ``.env`` file).
-
-2. **pix2tex** (local, MIT licence): crop each detected math bounding-box at
-   3× resolution and run the ``LatexOCR`` model locally.  Used when MathPix
-   credentials are absent.  Model (~116 MB) is downloaded on first call.
-
-The public entry point :func:`extract_formulas` routes to whichever backend
-is available.
-"""
-
-from __future__ import annotations
-
-import logging
-import unicodedata
-from pathlib import Path
-from typing import Any
-
-from codex.config import get_settings
-from codex.models import FormulaChunk
-
-logger = logging.getLogger(__name__)
-
-# Unicode categories that indicate mathematical symbols.
-_MATH_CATEGORIES: frozenset[str] = frozenset({"Sm", "So"})
-
-# Bbox size thresholds — avoids cropping tiny index fragments.
-_MIN_HEIGHT_PT = 15.0
-_MIN_WIDTH_PT = 40.0
-# Minimum number of math-category characters in a text block before we
-# consider it a formula candidate.
-_MIN_MATH_SYMBOLS = 5
-# Minimum ratio of math/total chars (as float 0–1).
-_MIN_MATH_RATIO = 0.15
-
-
-# ---------------------------------------------------------------------------
-# Internal helpers
-# ---------------------------------------------------------------------------
-
-
-def _is_math_char(ch: str) -> bool:
-    """Return True if *ch* belongs to a mathematical Unicode category."""
-    return unicodedata.category(ch) in _MATH_CATEGORIES
-
-
-def _math_ratio(text: str) -> float:
-    """Return the fraction of characters in *text* that are math chars."""
-    if not text:
-        return 0.0
-    math_count = sum(1 for ch in text if _is_math_char(ch))
-    return math_count / len(text)
-
-
-def _math_symbol_count(text: str) -> int:
-    """Return the absolute count of math-category characters in *text*."""
-    return sum(1 for ch in text if _is_math_char(ch))
-
-
-# ---------------------------------------------------------------------------
-# pix2tex singleton
-# ---------------------------------------------------------------------------
-
-_pix2tex_model: Any = None
-
-
-def _get_pix2tex_model() -> Any:
-    """Lazily load and return the pix2tex LatexOCR singleton."""
-    global _pix2tex_model  # noqa: PLW0603
-    if _pix2tex_model is None:
-        from pix2tex.cli import LatexOCR
-
-        _pix2tex_model = LatexOCR()
-    return _pix2tex_model
-
-
-# ---------------------------------------------------------------------------
-# pix2tex backend
-# ---------------------------------------------------------------------------
-
-
-def _extract_formulas_pix2tex(pdf_path: str) -> list[FormulaChunk]:
-    """Extract LaTeX formulas from *pdf_path* using pix2tex (local OCR).
-
-    Algorithm
-    ---------
-    1. Open the PDF with PyMuPDF (fitz).
-    2. For each page, gather text blocks; keep only blocks whose
-       math-symbol ratio exceeds the threshold and whose bounding box
-       is larger than the minimum size.
-    3. Crop the region at 3× resolution (better OCR quality).
-    4. Run pix2tex ``LatexOCR`` on the cropped image.
-    5. Wrap the result in a :class:`~codex.models.FormulaChunk`.
-    """
-    import fitz
-    from PIL import Image
-
-    results: list[FormulaChunk] = []
-    paper_id = Path(pdf_path).stem
-    model = _get_pix2tex_model()
-
-    try:
-        doc: Any = fitz.open(pdf_path)
-    except Exception:
-        logger.exception("fitz.open failed for %s", pdf_path)
-        return results
-
-    scale = fitz.Matrix(3, 3)
-
-    try:
-        for page_num in range(len(doc)):
-            page = doc[page_num]
-            blocks: list[Any] = page.get_text("blocks")
-
-            for block in blocks:
-                # block = (x0, y0, x1, y1, text, block_no, block_type)
-                if len(block) < 5:
-                    continue
-                x0, y0, x1, y1 = block[0], block[1], block[2], block[3]
-                text: str = block[4]
-
-                # Size filter
-                width_pt = x1 - x0
-                height_pt = y1 - y0
-                if width_pt < _MIN_WIDTH_PT or height_pt < _MIN_HEIGHT_PT:
-                    continue
-
-                # Math content filter
-                if (
-                    _math_symbol_count(text) < _MIN_MATH_SYMBOLS
-                    or _math_ratio(text) < _MIN_MATH_RATIO
-                ):
-                    continue
-
-                # Crop at 3× for OCR quality
-                clip_rect = fitz.Rect(x0, y0, x1, y1)
-                crop_pix = page.get_pixmap(matrix=scale, clip=clip_rect)
-                img = Image.frombytes(
-                    "RGB",
-                    (crop_pix.width, crop_pix.height),
-                    crop_pix.samples,
-                )
-
-                try:
-                    latex: str = model(img)
-                except Exception:
-                    logger.debug("pix2tex failed on page %d block, skipping", page_num + 1)
-                    continue
-
-                if not latex or not latex.strip():
-                    continue
-
-                context = text[:200].replace("\n", " ")
-                results.append(
-                    FormulaChunk(
-                        paper_id=paper_id,
-                        page=page_num + 1,
-                        raw_latex=latex.strip(),
-                        context=context,
-                    )
-                )
-    finally:
-        doc.close()
-
-    return results
-
-
-# ---------------------------------------------------------------------------
-# MathPix backend
-# ---------------------------------------------------------------------------
-
-
-def _extract_formulas_mathpix(
-    pdf_path: str,
-    app_id: str,
-    app_key: str,
-) -> list[FormulaChunk]:
-    """Extract LaTeX formulas using the MathPix v3 PDF API.
-
-    The function uploads the PDF, polls for results and parses the
-    returned ``latex_simplified`` list into :class:`~codex.models.FormulaChunk`
-    objects.
-
-    Retries with exponential back-off (via ``tenacity``) on transient HTTP
-    errors.
-    """
-    import httpx
-    from tenacity import retry, stop_after_attempt, wait_exponential
-
-    paper_id = Path(pdf_path).stem
-    headers = {
-        "app_id": app_id,
-        "app_key": app_key,
-    }
-
-    @retry(
-        stop=stop_after_attempt(4),
-        wait=wait_exponential(multiplier=1, min=2, max=30),
-        reraise=True,
-    )
-    def _post_pdf() -> dict[str, Any]:
-        with open(pdf_path, "rb") as fh:
-            resp = httpx.post(
-                "https://api.mathpix.com/v3/pdf",
-                headers=headers,
-                files={"file": fh},
-                timeout=60.0,
-            )
-        resp.raise_for_status()
-        return resp.json()  # type: ignore[no-any-return]
-
-    submit_response = _post_pdf()
-    pdf_id: str = submit_response.get("pdf_id", "")
-    if not pdf_id:
-        logger.warning("MathPix did not return a pdf_id — aborting")
-        return []
-
-    # Poll until complete
-    import time
-
-    for _ in range(60):
-        poll_resp = httpx.get(
-            f"https://api.mathpix.com/v3/pdf/{pdf_id}.mmd",
-            headers=headers,
-            timeout=30.0,
-        )
-        if poll_resp.status_code == 200:
-            break
-        time.sleep(2)
-    else:
-        logger.warning("MathPix polling timed out for pdf_id=%s", pdf_id)
-        return []
-
-    # Parse response — MathPix MMD contains \[ ... \] or $...$ blocks
-    mmd_text: str = poll_resp.text
-    results: list[FormulaChunk] = []
-    lines = mmd_text.splitlines()
-    in_block = False
-    block_lines: list[str] = []
-    page = 1
-
-    for line in lines:
-        if line.strip().startswith("\\["):
-            in_block = True
-            block_lines = [line]
-        elif in_block:
-            block_lines.append(line)
-            if line.strip().endswith("\\]"):
-                latex = "\n".join(block_lines).strip()
-                results.append(
-                    FormulaChunk(
-                        paper_id=paper_id,
-                        page=page,
-                        raw_latex=latex,
-                        context="",
-                    )
-                )
-                in_block = False
-                block_lines = []
-        # crude page tracking via MMD page markers
-        if line.strip().startswith("<!-- Page"):
-            page += 1
-
-    return results
-
-
-# ---------------------------------------------------------------------------
-# Public entry point
-# ---------------------------------------------------------------------------
-
-
-def extract_formulas(
-    pdf_path: str,
-    mathpix_app_id: str | None = None,
-    mathpix_app_key: str | None = None,
-) -> list[FormulaChunk]:
-    """Extract mathematical formulas from *pdf_path*.
-
-    Routes to the MathPix cloud backend when ``mathpix_app_id`` and
-    ``mathpix_app_key`` are both non-empty; otherwise falls back to
-    the local pix2tex model (if ``PIX2TEX_FALLBACK=true``, which is
-    the default).
-
-    Parameters
-    ----------
-    pdf_path:
-        Absolute or relative path to the PDF file.
-    mathpix_app_id:
-        Override for ``MATHPIX_APP_ID`` setting (optional).
-    mathpix_app_key:
-        Override for ``MATHPIX_APP_KEY`` setting (optional).
-
-    Returns
-    -------
-    list[FormulaChunk]
-        Extracted formulas; empty list if extraction is disabled or fails.
-    """
-    settings = get_settings()
-
-    settings_id = settings.mathpix_app_id.get_secret_value() if settings.mathpix_app_id else ""
-    settings_key = settings.mathpix_app_key.get_secret_value() if settings.mathpix_app_key else ""
-    app_id = mathpix_app_id or settings_id
-    app_key = mathpix_app_key or settings_key
-
-    if app_id and app_key:
-        logger.info("Using MathPix backend for %s", pdf_path)
-        return _extract_formulas_mathpix(pdf_path, app_id, app_key)
-
-    if settings.pix2tex_fallback:
-        logger.info("Using pix2tex fallback backend for %s", pdf_path)
-        return _extract_formulas_pix2tex(pdf_path)
-
-    logger.info("Formula extraction disabled (no creds, pix2tex_fallback=False)")
-    return []

codex/parsing/nougat.py

@@ -1,54 +0,0 @@
-"""Nougat OCR integration.
-
-Converts PDF files to Mathpix Markdown (mmd) by calling a self-hosted
-Nougat HTTP server.  No network fetching of papers happens here — the
-caller is expected to pass a local PDF path.
-"""
-
-from __future__ import annotations
-
-import httpx
-import tenacity
-
-from codex.config import Settings
-
-
-def pdf_to_markdown(pdf_path: str, nougat_url: str | None = None) -> str:
-    """Convert a local PDF to Mathpix Markdown via the Nougat HTTP API.
-
-    Parameters
-    ----------
-    pdf_path:
-        Absolute (or relative) path to the PDF file on disk.
-    nougat_url:
-        Base URL of the Nougat server.  Defaults to ``Settings().nougat_url``.
-
-    Returns
-    -------
-    str
-        The raw ``.mmd`` text returned by the server.
-
-    Raises
-    ------
-    httpx.HTTPStatusError
-        If the server returns a non-2xx status code.
-    """
-    if nougat_url is None:
-        nougat_url = Settings().nougat_url
-
-    @tenacity.retry(
-        retry=tenacity.retry_if_exception_type(httpx.ConnectError),
-        stop=tenacity.stop_after_attempt(3),  # 1 original + 2 retries
-        wait=tenacity.wait_fixed(0),
-        reraise=True,
-    )
-    def _post() -> str:
-        with open(pdf_path, "rb") as fh, httpx.Client(timeout=120.0) as client:
-            response = client.post(
-                f"{nougat_url}/predict",
-                files={"file": (pdf_path, fh, "application/pdf")},
-            )
-        response.raise_for_status()
-        return response.text
-
-    return _post()

codex/parsing/tex.py

@@ -1,223 +0,0 @@
-"""LaTeX parsing utilities.
-
-Provides helpers to extract sections, chunk text, and convert LaTeX to plain
-readable prose by stripping markup that is not useful for NLP/embedding.
-"""
-
-from __future__ import annotations
-
-import re
-
-# ---------------------------------------------------------------------------
-# Internal helpers
-# ---------------------------------------------------------------------------
-
-_SECTION_RE = re.compile(
-    r"\\(?:sub)*section\*?\s*\{([^}]*)\}",
-    re.DOTALL,
-)
-
-# \input{file} / \include{file} — arXiv papers routinely split the body across
-# files, so these must be inlined before parsing (R-C multi-file flattening).
-_INPUT_RE = re.compile(r"\\(?:input|include)\s*\{([^}]+)\}")
-
-# Patterns for LaTeX noise removal (applied in order).
-_COMMENT_RE = re.compile(r"%[^\n]*")
-_CITE_RE = re.compile(r"\\cite\*?\{[^}]*\}")
-_LABEL_RE = re.compile(r"\\label\{[^}]*\}")
-_REF_RE = re.compile(r"\\ref\{[^}]*\}")
-# Display math: $$...$$  (before single $ to avoid greedy mismatch)
-_DISPLAY_DOLLAR_RE = re.compile(r"\$\$.*?\$\$", re.DOTALL)
-# Inline math: $...$
-_INLINE_MATH_RE = re.compile(r"\$[^$\n]*?\$", re.DOTALL)
-# \begin{equation}...\end{equation}
-_ENV_EQUATION_RE = re.compile(
-    r"\\begin\{equation\*?\}.*?\\end\{equation\*?\}",
-    re.DOTALL,
-)
-# \begin{align}...\end{align}  (covers align, align*, aligned, …)
-_ENV_ALIGN_RE = re.compile(
-    r"\\begin\{align[^}]*\}.*?\\end\{align[^}]*\}",
-    re.DOTALL,
-)
-# Collapse excess whitespace
-_WHITESPACE_RE = re.compile(r"[ \t]+")
-_BLANK_LINES_RE = re.compile(r"\n{3,}")
-
-
-def _clean_latex(text: str) -> str:
-    """Strip LaTeX markup and return readable prose."""
-    text = _COMMENT_RE.sub("", text)
-    text = _ENV_EQUATION_RE.sub("", text)
-    text = _ENV_ALIGN_RE.sub("", text)
-    text = _DISPLAY_DOLLAR_RE.sub("", text)
-    text = _INLINE_MATH_RE.sub("", text)
-    text = _CITE_RE.sub("", text)
-    text = _LABEL_RE.sub("", text)
-    text = _REF_RE.sub("", text)
-    text = _WHITESPACE_RE.sub(" ", text)
-    text = _BLANK_LINES_RE.sub("\n\n", text)
-    return text.strip()
-
-
-# ---------------------------------------------------------------------------
-# Public API
-# ---------------------------------------------------------------------------
-
-
-def extract_sections(latex: str) -> list[tuple[str, str]]:
-    """Split *latex* on \\section / \\subsection boundaries.
-
-    Returns a list of ``(title, cleaned_text)`` tuples, one per section.
-    Text between the document start and the first section command is
-    discarded (preamble / abstract handling is out of scope).
-    """
-    # Find all section positions
-    matches = list(_SECTION_RE.finditer(latex))
-    if not matches:
-        return []
-
-    sections: list[tuple[str, str]] = []
-    for idx, match in enumerate(matches):
-        title = match.group(1).strip()
-        body_start = match.end()
-        body_end = matches[idx + 1].start() if idx + 1 < len(matches) else len(latex)
-        body = latex[body_start:body_end]
-        cleaned = _clean_latex(body)
-        sections.append((title, cleaned))
-
-    return sections
-
-
-def chunk_text(text: str, size: int = 512, overlap: int = 64) -> list[str]:
-    """Split *text* into overlapping word-based chunks.
-
-    Parameters
-    ----------
-    text:
-        Plain text to chunk (not raw LaTeX).
-    size:
-        Target number of words per chunk.
-    overlap:
-        Number of words to carry over into the next chunk.
-
-    Each chunk is at most ``size + overlap`` words.  Boundaries are snapped
-    to the nearest ``". "`` (sentence end) within ±20 words of the nominal
-    boundary when possible.
-    """
-    words = text.split()
-    if not words:
-        return []
-
-    snap_window = 20
-    chunks: list[str] = []
-    start = 0
-
-    while start < len(words):
-        end = min(start + size, len(words))
-
-        # Try to snap *end* to a sentence boundary within ±snap_window words.
-        if end < len(words):
-            best = end
-            # Build a small search window
-            lo = max(start + 1, end - snap_window)
-            hi = min(len(words), end + snap_window + 1)
-            # Prefer the closest sentence-end ". " to *end*
-            for offset in range(0, snap_window + 1):
-                for candidate in (end - offset, end + offset):
-                    if (
-                        lo <= candidate < hi
-                        and candidate > start
-                        and words[candidate - 1].endswith(".")
-                    ):
-                        # Sentence boundary: word at (candidate-1) ends with ".".
-                        best = candidate
-                        break
-                if best != end:
-                    break
-            end = best
-
-        chunk_words = words[start:end]
-        chunks.append(" ".join(chunk_words))
-
-        if end >= len(words):
-            break
-
-        # Next chunk starts *overlap* words before *end*.
-        start = max(start + 1, end - overlap)
-
-    return chunks
-
-
-def latex_to_text(latex: str) -> str:
-    """Convert a LaTeX document to plain text.
-
-    Extracts all sections, cleans each one, and joins them with a blank line.
-    If no section commands are found, the whole document is cleaned and
-    returned as a single block.
-    """
-    sections = extract_sections(latex)
-    if sections:
-        return "\n\n".join(body for _, body in sections)
-    # Fallback: no section structure — clean the whole string.
-    return _clean_latex(latex)
-
-
-def _norm_texkey(name: str) -> str:
-    """Normalize a tex file name / ``\\input`` argument to a comparable key.
-
-    Strips a leading ``./``, any directory-irrelevant whitespace, and a trailing
-    ``.tex`` so ``\\input{sections/intro}`` matches the archive member
-    ``sections/intro.tex``.
-    """
-    n = name.strip()
-    if n.startswith("./"):  # a single leading "./" only — don't eat "../" or ".hidden"
-        n = n[2:]
-    return n[:-4] if n.endswith(".tex") else n
-
-
-def flatten_inputs(primary: str, files: dict[str, str]) -> str:
-    """Recursively inline ``\\input``/``\\include`` directives (R-C).
-
-    *files* maps each source file's archive name to its contents (keys may carry
-    a ``.tex`` suffix or a ``./`` prefix — they are normalized internally). arXiv
-    multi-file projects leave the ``\\documentclass`` file as little more than a
-    skeleton of ``\\input`` lines, so the body must be assembled before parsing.
-    Unresolved references are dropped; recursion is capped at depth 10 to guard
-    against include cycles.
-    """
-    norm = {_norm_texkey(k): v for k, v in files.items()}
-
-    def _expand(text: str, depth: int) -> str:
-        if depth > 10:
-            # Cap reached (likely an \input cycle): strip any still-unresolved
-            # directives rather than leak a literal "\input{…}" into the parsed text.
-            return _INPUT_RE.sub("", text)
-
-        def _repl(match: re.Match[str]) -> str:
-            content = norm.get(_norm_texkey(match.group(1)))
-            return _expand(content, depth + 1) if content is not None else ""
-
-        return _INPUT_RE.sub(_repl, text)
-
-    return _expand(primary, 0)
-
-
-def chunk_sections(latex: str, size: int = 512, overlap: int = 64) -> list[tuple[str | None, str]]:
-    """Chunk LaTeX *within* each section, pairing every chunk with its title.
-
-    Unlike :func:`latex_to_text` (which discards titles and flattens the whole
-    document into one block before word-window chunking), this keeps chunks from
-    spanning section boundaries and remembers which ``\\section`` each came from.
-    The caller can then label the stored ``section`` column from the real heading
-    instead of mostly ``body`` (R-C / audit R-12). Falls back to title-less
-    chunks of the whole cleaned document when there are no section commands.
-    """
-    sections = extract_sections(latex)
-    if not sections:
-        return [(None, chunk) for chunk in chunk_text(_clean_latex(latex), size, overlap)]
-    pairs: list[tuple[str | None, str]] = []
-    for title, body in sections:
-        for chunk in chunk_text(body, size, overlap):
-            pairs.append((title or None, chunk))
-    return pairs

codex/provenance.py

@@ -1,184 +0,0 @@
-"""Provenance tracking: @cite-scan, code_links CRUD, references.bib export."""
-
-from __future__ import annotations
-
-import re
-from pathlib import Path
-
-from codex.db import get_conn
-from codex.models import CodeLink
-
-# ---------------------------------------------------------------------------
-# @cite scanning
-# ---------------------------------------------------------------------------
-
-_CITE_RE = re.compile(r"@cite\s+(\S+)", re.IGNORECASE)
-_FUNC_RE = re.compile(r"(\w+)\s*\(")
-_CLASS_RE = re.compile(r"class\s+(\w+)")
-
-_CPP_SUFFIXES = {".cpp", ".h", ".cxx", ".hpp"}
-
-
-def scan_cite_tags(root_dir: str) -> list[dict[str, str]]:
-    """Recursively scan C++ source files for @cite tags (Doxygen-style).
-
-    Returns one dict per hit:
-        file   (str) — relative path to the source file
-        line   (int as str) — 1-based line number
-        symbol (str) — function or class name (extracted from context, best-effort)
-        bibkey (str) — the @cite argument
-
-    Scans files matching: *.cpp, *.h, *.cxx, *.hpp
-    Pattern matched: ``@cite <bibkey>`` (case-insensitive, optional whitespace)
-    """
-    root = Path(root_dir)
-    results: list[dict[str, str]] = []
-
-    for path in sorted(root.rglob("*")):
-        if path.suffix.lower() not in _CPP_SUFFIXES:
-            continue
-        try:
-            lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
-        except OSError:
-            continue
-
-        rel_path = str(path.relative_to(root))
-
-        for lineno, line in enumerate(lines, start=1):
-            for match in _CITE_RE.finditer(line):
-                bibkey = match.group(1)
-                # Best-effort symbol extraction: scan backwards for func or class
-                symbol = ""
-                context_lines = lines[: lineno - 1]
-                for prev_line in reversed(context_lines):
-                    func_match = _FUNC_RE.search(prev_line)
-                    class_match = _CLASS_RE.search(prev_line)
-                    if func_match:
-                        symbol = func_match.group(1)
-                        break
-                    if class_match:
-                        symbol = class_match.group(1)
-                        break
-
-                results.append(
-                    {
-                        "file": rel_path,
-                        "line": str(lineno),
-                        "symbol": symbol,
-                        "bibkey": bibkey,
-                    }
-                )
-
-    return results
-
-
-# ---------------------------------------------------------------------------
-# code_links CRUD
-# ---------------------------------------------------------------------------
-
-
-def add_code_link(
-    symbol: str,
-    paper_id: str | None = None,
-    role: str | None = None,
-    note: str | None = None,
-) -> CodeLink:
-    """Insert a new code_link row and return the populated CodeLink (with db-assigned id)."""
-    sql = """
-        INSERT INTO code_links (symbol, paper_id, role, note)
-        VALUES (%(symbol)s, %(paper_id)s, %(role)s, %(note)s)
-        RETURNING id, added_at
-    """
-    with get_conn() as conn:
-        row = conn.execute(
-            sql,
-            {"symbol": symbol, "paper_id": paper_id, "role": role, "note": note},
-        ).fetchone()
-        conn.commit()
-    assert row is not None
-    return CodeLink(
-        symbol=symbol,
-        paper_id=paper_id,
-        role=role,
-        note=note,
-        id=row["id"],
-        added_at=row["added_at"],
-    )
-
-
-def list_code_links(paper_id: str | None = None) -> list[CodeLink]:
-    """Return all code_links, optionally filtered by paper_id."""
-    if paper_id is not None:
-        sql = (
-            "SELECT id, symbol, paper_id, role, note, added_at"
-            " FROM code_links WHERE paper_id = %(paper_id)s"
-        )
-        params: dict[str, str] = {"paper_id": paper_id}
-    else:
-        sql = "SELECT id, symbol, paper_id, role, note, added_at FROM code_links"
-        params = {}
-
-    with get_conn() as conn:
-        rows = conn.execute(sql, params).fetchall()
-
-    return [
-        CodeLink(
-            symbol=row["symbol"],
-            paper_id=row["paper_id"],
-            role=row["role"],
-            note=row["note"],
-            id=row["id"],
-            added_at=row["added_at"],
-        )
-        for row in rows
-    ]
-
-
-# ---------------------------------------------------------------------------
-# references.bib export
-# ---------------------------------------------------------------------------
-
-
-def export_bib(paper_ids: list[str] | None = None) -> str:
-    """Export papers as a BibTeX string.
-
-    If paper_ids is None, exports ALL papers with a non-NULL bibkey.
-    Only papers with a non-NULL bibkey are included (bibkey is the BibTeX key).
-
-    Entry format (article):
-        @article{<bibkey>,
-          title  = {<title>},
-          author = {<authors joined with " and ">},
-          year   = {<year>},
-          note   = {<id>},
-        }
-    """
-    if paper_ids is not None:
-        sql = (
-            "SELECT id, bibkey, title, authors, year"
-            " FROM papers"
-            " WHERE bibkey IS NOT NULL AND id = ANY(%(paper_ids)s)"
-        )
-        params: dict[str, list[str]] = {"paper_ids": paper_ids}
-    else:
-        sql = "SELECT id, bibkey, title, authors, year FROM papers WHERE bibkey IS NOT NULL"
-        params = {}
-
-    with get_conn() as conn:
-        rows = conn.execute(sql, params).fetchall()
-
-    entries: list[str] = []
-    for row in rows:
-        authors: list[str] = row["authors"] or []
-        author_str = " and ".join(authors) if authors else ""
-        entry = (
-            f"@article{{{row['bibkey']},\n"
-            f"  title  = {{{row['title']}}},\n"
-            f"  author = {{{author_str}}},\n"
-            f"  year   = {{{row['year']}}},\n"
-            f"  note   = {{{row['id']}}},\n"
-            f"}}"
-        )
-        entries.append(entry)
-
-    return "\n\n".join(entries)

codex/quality.py

@@ -1,248 +0,0 @@
-"""Chunk quality filtering and section classification (F-16).
-
-Three independent quality signals are applied at ingest time:
-
-1. **Length** — structural; no content required.
-2. **Alpha-ratio** — OCR artefacts have high non-alpha character density.
-3. **Bib-score** — DOI + "et al." + (YYYY) patterns co-occur almost only
-   in reference list entries.
-
-Section classification is rule-based (no LLM) and runs on the first 200
-characters of each chunk.  The retroactive ``run_quality_pass`` function
-can be called from the CLI to back-fill existing chunks.
-"""
-
-from __future__ import annotations
-
-import logging
-import re
-from typing import TYPE_CHECKING, Any
-
-if TYPE_CHECKING:
-    from codex.config import Settings
-
-logger = logging.getLogger(__name__)
-
-
-# ---------------------------------------------------------------------------
-# Section classification patterns (checked in order; first match wins)
-# ---------------------------------------------------------------------------
-
-_SECTION_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
-    ("abstract", re.compile(r"^\s*(abstract|zusammenfassung|r[eé]sum[eé])\b", re.I)),
-    ("intro", re.compile(r"^\s*(introduction|einleitung|1\.\s)", re.I)),
-    ("theorem", re.compile(r"^\s*(theorem|lemma|proposition|corollary|definition)\b", re.I)),
-    ("proof", re.compile(r"^\s*(proof\b|beweis\b|proof\s+of\b)", re.I)),
-    ("bibliography", re.compile(r"^\s*(references|bibliography|bibliographie|literatur)\b", re.I)),
-]
-
-_DOI_RE = re.compile(r"10\.\d{4,}/\S+")
-_ET_AL_RE = re.compile(r"\bet\s+al\b", re.I)
-_YEAR_BRACKETS_RE = re.compile(r"\(\d{4}\)")
-
-
-# ---------------------------------------------------------------------------
-# Bibliography heuristic score
-# ---------------------------------------------------------------------------
-
-
-def _bib_score(text: str) -> float:
-    """Return a [0..1] heuristic for how bibliography-like a chunk is.
-
-    Three signals contribute:
-    * DOI occurrences (weight 3)
-    * "et al." occurrences (weight 2)
-    * year-in-brackets occurrences (weight 1)
-
-    The raw signal is normalised against a rough word-count proxy so that
-    long chunks with occasional references don't get flagged.
-    """
-    if not text:
-        return 0.0
-    word_count = max(len(text.split()), 1)
-    doi_hits = len(_DOI_RE.findall(text))
-    etal_hits = len(_ET_AL_RE.findall(text))
-    year_hits = len(_YEAR_BRACKETS_RE.findall(text))
-    raw = (doi_hits * 3 + etal_hits * 2 + year_hits) / max(word_count / 10, 1)
-    return min(raw, 1.0)
-
-
-# ---------------------------------------------------------------------------
-# Quality predicate
-# ---------------------------------------------------------------------------
-
-
-def is_quality_chunk(text: str, *, settings: Settings) -> bool:
-    """Return True when *text* passes all three quality thresholds.
-
-    Checks (all configurable via :class:`codex.config.Settings`):
-
-    1. ``chunk_min_chars`` — character count floor.
-    2. ``chunk_min_alpha_ratio`` — minimum fraction of alphabetic chars.
-    3. ``chunk_max_bib_score`` — bibliography heuristic ceiling.
-    """
-    if len(text) < settings.chunk_min_chars:
-        return False
-    alpha_ratio = sum(c.isalpha() for c in text) / max(len(text), 1)
-    if alpha_ratio < settings.chunk_min_alpha_ratio:
-        return False
-    return _bib_score(text) <= settings.chunk_max_bib_score
-
-
-# ---------------------------------------------------------------------------
-# Section classification
-# ---------------------------------------------------------------------------
-
-
-def classify_section(text: str) -> str:
-    """Classify a chunk's section using rule-based regex matching.
-
-    Inspects only the first 200 characters.  Returns one of:
-    ``abstract``, ``intro``, ``theorem``, ``proof``, ``bibliography``, ``body``.
-    """
-    snippet = text[:200]
-    for section_name, pattern in _SECTION_PATTERNS:
-        if pattern.search(snippet):
-            return section_name
-    # Fallback: bibliography by DOI density even without a header
-    if _bib_score(text) > 0.5:
-        return "bibliography"
-    return "body"
-
-
-def _clean_title(title: str) -> str:
-    """Normalize a LaTeX ``\\section`` title for use as a section label.
-
-    Strips LaTeX word-commands (``\\emph`` …), accent/escape macros (``\\"o`` →
-    ``o``, ``\\&``), ``~`` ties and ``{}$``, drops leading section numbering
-    (``3.2 ``), collapses whitespace, lower-cases, and truncates to 60 chars at a
-    word boundary so the stored ``section`` value is a clean, comparable string.
-    """
-    t = title.replace("~", " ")  # LaTeX non-breaking tie → space
-    t = re.sub(r"\\[a-zA-Z]+\*?", " ", t)  # word-commands: \emph, \mathcal …
-    t = re.sub(r"\\[^a-zA-Z]", "", t)  # accent/escape macros: \"o → o, \', \`, \&
-    t = re.sub(r"[{}$]", "", t)  # braces / math delimiters
-    t = re.sub(r"^\s*\d+(?:\.\d+)*\.?\s*", "", t)  # leading numbering "3.2 "
-    t = " ".join(t.split()).strip().lower()
-    return t[:60].rsplit(" ", 1)[0] if len(t) > 60 else t
-
-
-# A heading collapses to a controlled bucket only when it IS one of these canonical
-# section words — NOT when it merely starts with one. Prefix-matching via
-# classify_section mislabels real sections ("Abstract Nonsense and Categories" →
-# abstract, "References Architecture" → bibliography); an exact map avoids that while
-# keeping cross-source consistency for the bare headings.
-_SECTION_TITLE_BUCKETS = {
-    "abstract": "abstract",
-    "zusammenfassung": "abstract",
-    "introduction": "intro",
-    "einleitung": "intro",
-    "proof": "proof",
-    "beweis": "proof",
-    "references": "bibliography",
-    "bibliography": "bibliography",
-    "bibliographie": "bibliography",
-    "literatur": "bibliography",
-}
-
-
-def section_label(title: str | None, content: str) -> str:
-    """Label a chunk's section, preferring the real ``\\section`` heading (R-F).
-
-    For section-aware ``.tex`` ingest *title* is the real heading: a bare canonical
-    heading ("Introduction", "References", "Proof") maps to its controlled bucket
-    (cross-source consistency); any other heading is stored as its cleaned real
-    title (e.g. ``"preliminaries"``, ``"introduction to operator algebras"``) — far
-    more signal than collapsing to ``body`` AND without the prefix-match mislabels
-    that ``classify_section`` would produce on descriptive titles.
-
-    When *title* is None / blank (the ``.pdf`` / ``.txt`` / ``run_quality_pass``
-    paths) this falls back to content-based :func:`classify_section` — unchanged
-    behaviour. Safe to store free-text titles because the ``section`` column is
-    write-only (no consumer filters on the controlled vocabulary).
-    """
-    if not title or not title.strip():
-        return classify_section(content)
-    cleaned = _clean_title(title)
-    if not cleaned:
-        return classify_section(content)
-    return _SECTION_TITLE_BUCKETS.get(cleaned, cleaned)
-
-
-# ---------------------------------------------------------------------------
-# Batch filter
-# ---------------------------------------------------------------------------
-
-
-def filter_chunks(chunks: list[str], *, settings: Settings) -> list[str]:
-    """Return only the chunks that pass all quality filters.
-
-    Logs the keep ratio at DEBUG level.
-    """
-    kept = [c for c in chunks if is_quality_chunk(c, settings=settings)]
-    logger.debug(
-        "Quality filter: %d/%d chunks kept (%.0f%%)",
-        len(kept),
-        len(chunks),
-        100 * len(kept) / max(len(chunks), 1),
-    )
-    return kept
-
-
-# ---------------------------------------------------------------------------
-# Retroactive DB pass
-# ---------------------------------------------------------------------------
-
-
-def run_quality_pass(
-    *,
-    paper_id: str | None = None,
-    conn: Any,
-    settings: Settings,
-) -> dict[str, int]:
-    """Apply quality filter + section classification to existing chunks in DB.
-
-    For each chunk:
-    * Fails quality → DELETE.
-    * Passes quality → UPDATE ``section`` with :func:`classify_section`.
-
-    Parameters
-    ----------
-    paper_id:
-        When provided, restrict the pass to chunks for this paper only.
-    conn:
-        Open psycopg connection (dict-row factory assumed).
-    settings:
-        Application settings for quality thresholds.
-
-    Returns
-    -------
-    dict with keys ``kept``, ``removed``, ``tagged`` (= kept).
-    """
-    if paper_id is not None:
-        rows = conn.execute(
-            "SELECT id, content FROM chunks WHERE paper_id = %(pid)s",
-            {"pid": paper_id},
-        ).fetchall()
-    else:
-        rows = conn.execute("SELECT id, content FROM chunks").fetchall()
-
-    kept = 0
-    removed = 0
-    for row in rows:
-        chunk_id = row["id"]
-        content = row["content"]
-        if not is_quality_chunk(content, settings=settings):
-            conn.execute("DELETE FROM chunks WHERE id = %(id)s", {"id": chunk_id})
-            removed += 1
-        else:
-            section = classify_section(content)
-            conn.execute(
-                "UPDATE chunks SET section = %(section)s WHERE id = %(id)s",
-                {"section": section, "id": chunk_id},
-            )
-            kept += 1
-
-    conn.commit()
-    logger.info("Quality pass done: %d kept, %d removed, %d section-tagged", kept, removed, kept)
-    return {"kept": kept, "removed": removed, "tagged": kept}

codex/sources/arxiv.py

@@ -1,206 +0,0 @@
-"""arXiv API client.
-
-Provides:
-- fetch_metadata: resolve an arXiv ID to a Paper via the Atom export API.
-- fetch_source: download the .tar.gz source of a paper and extract the primary .tex file.
-- fetch_pdf_url: return the canonical PDF URL for a given arXiv ID.
-"""
-
-from __future__ import annotations
-
-import gzip
-import io
-import logging
-import tarfile
-import xml.etree.ElementTree as ET
-
-import httpx
-from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential
-
-from codex.models import Paper
-
-logger = logging.getLogger(__name__)
-
-_BASE = "https://arxiv.org"
-_EXPORT = "https://export.arxiv.org"
-_ATOM = "{http://www.w3.org/2005/Atom}"
-
-
-def _is_retryable(exc: BaseException) -> bool:
-    if isinstance(exc, httpx.HTTPStatusError):
-        return exc.response.status_code == 429 or exc.response.status_code >= 500
-    return False
-
-
-@retry(
-    retry=retry_if_exception(_is_retryable),
-    stop=stop_after_attempt(4),
-    wait=wait_exponential(min=1, max=20),
-    before_sleep=lambda rs: logger.warning("arXiv retry %d", rs.attempt_number),
-)
-def _query(arxiv_id: str) -> httpx.Response:
-    response = httpx.get(
-        f"{_EXPORT}/api/query",
-        params={"id_list": arxiv_id},
-        timeout=30,
-        follow_redirects=True,
-    )
-    response.raise_for_status()
-    return response
-
-
-def fetch_metadata(arxiv_id: str) -> Paper | None:
-    """Resolve an arXiv ID to a Paper via the Atom export API.
-
-    Authoritative metadata source for arXiv preprints — used as an ingest
-    fallback when OpenAlex 404s on an arXiv id (DQ-2). The arXiv id (bare,
-    e.g. ``"1911.00966"`` or legacy ``"math/0603097"``) becomes ``Paper.id``;
-    ``openalex_id`` and ``bibkey`` are left for the caller to populate.
-
-    Returns
-    -------
-    Paper | None
-        Populated Paper (title, authors, year, abstract), or None if arXiv
-        has no entry for the id.
-    """
-    bare = arxiv_id[len("arxiv:") :] if arxiv_id.lower().startswith("arxiv:") else arxiv_id
-    try:
-        response = _query(bare)
-    except httpx.HTTPError:
-        logger.warning("arXiv metadata fetch failed for %s", bare, exc_info=True)
-        return None
-    try:
-        root = ET.fromstring(response.text)
-    except ET.ParseError:
-        return None
-    entry = root.find(f"{_ATOM}entry")
-    if entry is None:
-        return None
-    # An id-not-found query still returns a feed but with no <entry>.
-    title = (entry.findtext(f"{_ATOM}title") or "").strip()
-    if not title:
-        return None
-    summary = (entry.findtext(f"{_ATOM}summary") or "").strip()
-    published = entry.findtext(f"{_ATOM}published") or ""
-    year = int(published[:4]) if published[:4].isdigit() else None
-    authors = [
-        name.strip()
-        for a in entry.findall(f"{_ATOM}author")
-        if (name := a.findtext(f"{_ATOM}name")) and name.strip()
-    ]
-    return Paper(
-        id=bare,
-        title=" ".join(title.split()),
-        authors=authors,
-        year=year,
-        abstract=" ".join(summary.split()) or None,
-    )
-
-
-def _has_documentclass(content: str) -> bool:
-    """True if *content* has an UN-commented ``\\documentclass`` line.
-
-    A plain ``"\\documentclass" in content`` substring test also matches a
-    commented-out ``%\\documentclass`` (common in arXiv preambles); requiring the
-    command to start its line avoids selecting the wrong primary file.
-    """
-    return any(line.lstrip().startswith("\\documentclass") for line in content.splitlines())
-
-
-def fetch_source(arxiv_id: str) -> str | None:
-    """Download and extract the primary LaTeX source for an arXiv paper.
-
-    Downloads the .tar.gz source bundle from ``https://arxiv.org/src/{arxiv_id}``,
-    locates the primary .tex file (preferring any file containing ``\\documentclass``,
-    falling back to the largest .tex by size), and inlines its ``\\input``/
-    ``\\include`` directives from the other archive members so multi-file projects
-    return the full document body, not just the primary file's include skeleton.
-
-    Parameters
-    ----------
-    arxiv_id:
-        The arXiv identifier (e.g. ``"2301.07041"``).
-
-    Returns
-    -------
-    str | None
-        Raw LaTeX source string, or None if the paper is not found or no .tex
-        file is present (signals Nougat fallback).
-    """
-    url = f"{_BASE}/src/{arxiv_id}"
-    response = httpx.get(url, timeout=60, follow_redirects=True)
-    if response.status_code == 404:
-        logger.debug("arXiv 404 for source id=%s", arxiv_id)
-        return None
-    if response.status_code != 200:
-        response.raise_for_status()
-
-    from codex.parsing.tex import flatten_inputs
-
-    raw = response.content
-    # Image/font/binary members are never \input'd as text; skip them so the source
-    # map stays text-only (decode-with-replace would otherwise store garbled bytes).
-    skip_ext = (".png", ".jpg", ".jpeg", ".gif", ".bmp", ".pdf", ".eps", ".ps", ".ttf", ".otf")
-    try:
-        with tarfile.open(fileobj=io.BytesIO(raw), mode="r:gz") as tf:
-            files: dict[str, str] = {}
-            primary_name: str | None = None
-            for member in tf.getmembers():
-                if not member.isfile() or member.name.lower().endswith(skip_ext):
-                    continue
-                f = tf.extractfile(member)
-                if f is None:
-                    continue
-                # Keep ALL text members resolvable so \input of a non-.tex include
-                # (e.g. a .def or extension-less body file) is not silently dropped (R-C).
-                files[member.name] = f.read().decode("utf-8", errors="replace")
-                # Primary doc = a .tex with an UN-commented \documentclass.
-                if (
-                    primary_name is None
-                    and member.name.endswith(".tex")
-                    and _has_documentclass(files[member.name])
-                ):
-                    primary_name = member.name
-
-            tex_files = {k: v for k, v in files.items() if k.endswith(".tex")}
-            if not tex_files:
-                logger.debug("No .tex files found in arXiv source for %s", arxiv_id)
-                return None
-
-            # No \documentclass anywhere → fall back to the largest .tex.
-            if primary_name is None:
-                primary_name = max(tex_files, key=lambda k: len(tex_files[k]))
-
-            # Inline \input/\include so multi-file projects yield the full body,
-            # not just the primary file's skeleton of include directives (R-C).
-            return flatten_inputs(files[primary_name], files)
-    except tarfile.TarError:
-        # Old arXiv submissions are served as a single bare gzipped .tex with no
-        # tar wrapper (e.g. legacy math/* papers) — gunzip the body directly.
-        try:
-            single = gzip.decompress(raw).decode("utf-8", errors="replace")
-        except (OSError, EOFError) as exc:
-            logger.warning("Failed to read arXiv source for %s: %s", arxiv_id, exc)
-            return None
-        if "\\" in single[:5000]:  # looks like LaTeX (has backslash commands)
-            return single
-        logger.debug("arXiv source for %s is not LaTeX", arxiv_id)
-        return None
-
-
-def fetch_pdf_url(arxiv_id: str) -> str:
-    """Return the canonical PDF URL for an arXiv paper.
-
-    This is a pure computation — no HTTP request is made.
-
-    Parameters
-    ----------
-    arxiv_id:
-        The arXiv identifier (e.g. ``"2301.07041"``).
-
-    Returns
-    -------
-    str
-        The full URL of the PDF file.
-    """
-    return f"{_BASE}/pdf/{arxiv_id}.pdf"

codex/sources/crossref.py

@@ -1,126 +0,0 @@
-"""Crossref API client.
-
-Provides:
-- fetch_abstract: retrieve a work's abstract (JATS-stripped) by DOI.
-- fetch_references: retrieve a work's reference list (DOI edges) by DOI.
-
-Crossref is a *third* metadata source after OpenAlex and Semantic Scholar
-(DQ-2 abstracts / roadmap R-B references). Requests use the Polite Pool (mailto
-query parameter) and are retried on 429/5xx with exponential back-off via tenacity.
-"""
-
-from __future__ import annotations
-
-import logging
-import re
-from typing import Any
-
-import httpx
-from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential
-
-from codex.config import get_settings
-from codex.models import Citation
-
-logger = logging.getLogger(__name__)
-
-_BASE = "https://api.crossref.org"
-_TAG_RE = re.compile(r"<[^>]+>")
-
-
-def _is_retryable(exc: BaseException) -> bool:
-    if isinstance(exc, httpx.HTTPStatusError):
-        return exc.response.status_code == 429 or exc.response.status_code >= 500
-    return False
-
-
-def _polite_params() -> dict[str, str]:
-    # Reuse the OpenAlex mailto for Crossref's Polite Pool (same address).
-    mailto = get_settings().openalex_mailto
-    return {"mailto": mailto} if mailto else {}
-
-
-@retry(
-    retry=retry_if_exception(_is_retryable),
-    stop=stop_after_attempt(5),
-    wait=wait_exponential(min=1, max=30),
-    before_sleep=lambda rs: logger.warning(
-        "Crossref retry %d after %s",
-        rs.attempt_number,
-        rs.outcome.exception(),  # type: ignore[union-attr]
-    ),
-)
-def _get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-    merged: dict[str, str] = {**_polite_params(), **(params or {})}
-    response = httpx.get(url, params=merged, timeout=30, follow_redirects=True)
-    response.raise_for_status()
-    return response
-
-
-def _strip_jats(abstract: str | None) -> str | None:
-    """Strip JATS/XML tags from a Crossref abstract and collapse whitespace."""
-    if not abstract:
-        return None
-    text = _TAG_RE.sub("", abstract)
-    text = " ".join(text.split()).strip()
-    return text or None
-
-
-def fetch_abstract(doi: str) -> str | None:
-    """Fetch a work's abstract from Crossref by DOI.
-
-    Crossref stores abstracts as JATS-tagged markup (``<jats:p>…</jats:p>``);
-    tags are stripped before returning. Many works (notably book chapters)
-    have no abstract deposited — returns None in that case and on 404.
-
-    Parameters
-    ----------
-    doi:
-        A bare DOI (``"10.1007/s00454-019-00132-8"``).
-
-    Returns
-    -------
-    str | None
-        The plain-text abstract, or None when absent.
-    """
-    url = f"{_BASE}/works/{doi}"
-    try:
-        response = _get(url)
-    except httpx.HTTPStatusError as exc:
-        if exc.response.status_code == 404:
-            return None
-        raise
-    message: dict[str, Any] = response.json().get("message", {})
-    return _strip_jats(message.get("abstract"))
-
-
-def fetch_references(doi: str) -> list[Citation]:
-    """Fetch a work's reference list from Crossref by DOI (roadmap R-B).
-
-    Crossref carries publisher-deposited reference lists in ``message.reference``.
-    Each entry that cites a DOI-registered work exposes a bare ``DOI`` field;
-    only those become citation-graph edges (book / older references carry just
-    ``unstructured`` text with no resolvable id and are skipped). ``citing_id``
-    is the queried *doi* — the caller rewrites it to the canonical ``papers.id``
-    and normalises the cited DOIs (mirrors the Semantic Scholar reference path).
-
-    Parameters
-    ----------
-    doi:
-        A bare DOI (``"10.1007/s00454-019-00132-8"``).
-
-    Returns
-    -------
-    list[Citation]
-        One Citation per reference that carries a DOI. Empty on 404 or when no
-        references are deposited for the work.
-    """
-    url = f"{_BASE}/works/{doi}"
-    try:
-        response = _get(url)
-    except httpx.HTTPStatusError as exc:
-        if exc.response.status_code == 404:
-            return []
-        raise
-    message: dict[str, Any] = response.json().get("message", {})
-    references: list[dict[str, Any]] = message.get("reference") or []
-    return [Citation(citing_id=doi, cited_id=ref["DOI"]) for ref in references if ref.get("DOI")]

codex/sources/openalex.py

@@ -1,200 +0,0 @@
-"""OpenAlex API client.
-
-Provides:
-- fetch_paper: resolve an arXiv ID or DOI to a Paper dataclass.
-- fetch_citations: retrieve reference list as Citation dataclasses.
-
-All requests use the OpenAlex Polite Pool (mailto query parameter) and
-are retried on 429/5xx with exponential back-off via tenacity.
-"""
-
-from __future__ import annotations
-
-import logging
-from typing import Any
-
-import httpx
-from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential
-
-from codex.config import get_settings
-from codex.models import Citation, Paper
-
-logger = logging.getLogger(__name__)
-
-_BASE = "https://api.openalex.org"
-
-
-def _resolve_id(id: str) -> str:
-    """Resolve a bare DOI or arXiv ID for the OpenAlex /works/ endpoint.
-
-    OpenAlex resolves DOIs, W-IDs and full https:// URLs in the /works/ path,
-    but NOT the ``arxiv:`` namespace (that path 404s). arXiv papers are looked
-    up via their arXiv DOI ``10.48550/arXiv.<id>`` instead — accepted by
-    OpenAlex for both modern (``2301.07041``) and legacy (``math/0603097``) IDs.
-    """
-    s = id.strip()
-    if s.lower().startswith("arxiv:"):
-        s = s[len("arxiv:") :]
-    if s.startswith(("W", "https://", "doi:")):
-        return s
-    if s.startswith("10."):
-        return f"doi:{s}"
-    return f"doi:10.48550/arXiv.{s}"
-
-
-def _is_retryable(exc: BaseException) -> bool:
-    """Return True for HTTP 429 / 5xx responses wrapped in httpx.HTTPStatusError."""
-    if isinstance(exc, httpx.HTTPStatusError):
-        return exc.response.status_code == 429 or exc.response.status_code >= 500
-    return False
-
-
-def _polite_params() -> dict[str, str]:
-    mailto = get_settings().openalex_mailto
-    if mailto:
-        return {"mailto": mailto}
-    return {}
-
-
-@retry(
-    retry=retry_if_exception(_is_retryable),
-    stop=stop_after_attempt(5),
-    wait=wait_exponential(min=1, max=30),
-    before_sleep=lambda rs: logger.warning(
-        "OpenAlex retry %d after %s",
-        rs.attempt_number,
-        rs.outcome.exception(),  # type: ignore[union-attr]
-    ),
-)
-def _get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-    merged: dict[str, str] = {**_polite_params(), **(params or {})}
-    response = httpx.get(url, params=merged, timeout=30)
-    response.raise_for_status()
-    return response
-
-
-def _reconstruct_abstract(inverted_index: dict[str, list[int]] | None) -> str | None:
-    """Convert OpenAlex abstract_inverted_index back to plain text."""
-    if not inverted_index:
-        return None
-    word_positions: list[tuple[int, str]] = []
-    for word, positions in inverted_index.items():
-        for pos in positions:
-            word_positions.append((pos, word))
-    word_positions.sort(key=lambda t: t[0])
-    return " ".join(w for _, w in word_positions)
-
-
-def _normalize_doi(doi: str) -> str:
-    """Normalize an OpenAlex DOI to the canonical bare, lower-cased form.
-
-    OpenAlex returns the ``doi`` field as a full URL
-    (``https://doi.org/10.1007/s00454-019-00132-8``), but the canonical
-    ``papers.id`` produced by the M-1 migration is the BARE, lower-cased DOI
-    (``10.1007/s00454-019-00132-8``). Stripping the ``https://doi.org/`` /
-    ``doi:`` prefix and lower-casing (DOIs are case-insensitive by spec) makes a
-    re-ingested DOI paper match its stored row, so the ingest
-    ``INSERT … ON CONFLICT (id)`` updates in place instead of attempting a fresh
-    INSERT that trips the separate ``papers_openalex_id_key`` unique constraint
-    (DQ-5). Mirrors the cited-id convention in ``ingest._norm_cited_id``.
-    """
-    s = doi.strip().lower()
-    for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
-        if s.startswith(prefix):
-            return s[len(prefix) :]
-    return s
-
-
-# arXiv works are registered under a DOI of the form 10.48550/arXiv.<id>.
-_ARXIV_DOI_PREFIX = "10.48550/arxiv."
-
-
-def _canonical_id(doi: str) -> str:
-    """Return the canonical ``papers.id`` for an OpenAlex ``doi`` value.
-
-    Builds on :func:`_normalize_doi` (bare, lower-cased DOI), then strips the
-    arXiv DOI prefix ``10.48550/arxiv.`` so an arXiv work's id is the BARE arXiv
-    id (``2305.10988`` / ``math/0603097``) — the form the corpus is keyed on —
-    not the arXiv DOI. Without this, re-ingesting an arXiv paper by its bare id
-    misses ``INSERT … ON CONFLICT (id)`` and trips ``papers_openalex_id_key``.
-    This is the arXiv half of DQ-5 (the DOI half was fixed there; this was
-    explicitly deferred and surfaced again driving the R-C .tex re-ingest).
-    """
-    normalized = _normalize_doi(doi)
-    if normalized.startswith(_ARXIV_DOI_PREFIX):
-        return normalized[len(_ARXIV_DOI_PREFIX) :]
-    return normalized
-
-
-def _map_paper(data: dict[str, Any]) -> Paper:
-    authors: list[str] = [
-        a.get("author", {}).get("display_name") or "" for a in data.get("authorships", [])
-    ]
-    abstract = _reconstruct_abstract(data.get("abstract_inverted_index"))
-    doi = data.get("doi")
-    return Paper(
-        id=_canonical_id(doi) if doi else (data.get("id") or ""),
-        title=data.get("title") or "",
-        openalex_id=data.get("id"),
-        authors=authors,
-        year=data.get("publication_year"),
-        abstract=abstract,
-    )
-
-
-def fetch_paper(id: str) -> Paper | None:
-    """Fetch a single paper by arXiv ID or DOI.
-
-    Parameters
-    ----------
-    id:
-        An arXiv ID (``"2301.07041"``), a DOI (``"10.1145/…"``),
-        or an OpenAlex W-ID (``"W2741809807"``).
-
-    Returns
-    -------
-    Paper | None
-        A populated Paper instance, or None on 404.
-    """
-    url = f"{_BASE}/works/{_resolve_id(id)}"
-    try:
-        response = _get(url)
-    except httpx.HTTPStatusError as exc:
-        if exc.response.status_code == 404:
-            logger.debug("OpenAlex 404 for id=%s", id)
-            return None
-        raise
-    return _map_paper(response.json())
-
-
-def fetch_citations(openalex_id: str) -> list[Citation]:
-    """Fetch the reference list for a paper as Citation objects.
-
-    Uses the ``referenced_works`` field from the work object — the
-    ``/works/{id}/references`` path is not a real OpenAlex endpoint.
-
-    Parameters
-    ----------
-    openalex_id:
-        The OpenAlex work ID (``"https://openalex.org/W…"`` or ``"W…"``).
-
-    Returns
-    -------
-    list[Citation]
-        One Citation per referenced work (OpenAlex W-ID as cited_id).
-    """
-    url = f"{_BASE}/works/{_resolve_id(openalex_id)}"
-    try:
-        response = _get(url)
-    except httpx.HTTPStatusError as exc:
-        if exc.response.status_code == 404:
-            return []
-        raise
-
-    data: dict[str, Any] = response.json()
-    referenced_works: list[str] = data.get("referenced_works", [])
-    return [
-        Citation(citing_id=openalex_id, cited_id=cited_id)
-        for cited_id in referenced_works
-        if cited_id
-    ]

codex/sources/semanticscholar.py

@@ -1,157 +0,0 @@
-"""Semantic Scholar API client.
-
-Provides:
-- fetch_references: retrieve references for a paper as Citation dataclasses.
-- fetch_recommendations: retrieve recommended paper IDs.
-
-Rate-limited to ≤1 req/s (per-request floor via monotonic clock).
-Retried on 429/5xx with exponential back-off via tenacity.
-"""
-
-from __future__ import annotations
-
-import logging
-import threading
-import time
-from typing import Any
-from urllib.parse import quote
-
-import httpx
-from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential
-
-from codex.models import Citation
-
-logger = logging.getLogger(__name__)
-
-_BASE_GRAPH = "https://api.semanticscholar.org/graph/v1"
-_BASE_RECS = "https://api.semanticscholar.org/recommendations/v1"
-
-# Per-request rate limit: ≤1 req/s without an API key.
-_rate_lock = threading.Lock()
-_last_request_time: float = 0.0
-_MIN_INTERVAL = 1.0
-
-
-def _rate_limit() -> None:
-    global _last_request_time
-    with _rate_lock:
-        now = time.monotonic()
-        wait = _MIN_INTERVAL - (now - _last_request_time)
-        if wait > 0:
-            time.sleep(wait)
-        _last_request_time = time.monotonic()
-
-
-def _is_retryable(exc: BaseException) -> bool:
-    if isinstance(exc, httpx.HTTPStatusError):
-        return exc.response.status_code == 429 or exc.response.status_code >= 500
-    return False
-
-
-@retry(
-    retry=retry_if_exception(_is_retryable),
-    stop=stop_after_attempt(5),
-    wait=wait_exponential(min=1, max=30),
-    before_sleep=lambda rs: logger.warning(
-        "SemanticScholar retry %d after %s",
-        rs.attempt_number,
-        rs.outcome.exception(),  # type: ignore[union-attr]
-    ),
-)
-def _get(url: str, params: dict[str, Any] | None = None) -> httpx.Response:
-    _rate_limit()
-    response = httpx.get(url, params=params, timeout=30)
-    response.raise_for_status()
-    return response
-
-
-def fetch_references(paper_id: str) -> list[Citation]:
-    """Fetch references for a paper from Semantic Scholar.
-
-    Parameters
-    ----------
-    paper_id:
-        Semantic Scholar paper ID (or ``arXiv:…`` / ``DOI:…`` prefixed ID).
-
-    Returns
-    -------
-    list[Citation]
-        One Citation per reference, with optional context snippet.
-    """
-    # quote the id so a legacy-arXiv "/" (e.g. arXiv:math/0603097) doesn't break
-    # the URL path; keep ":" for the arXiv:/DOI: scheme prefix (audit R-5).
-    url = f"{_BASE_GRAPH}/paper/{quote(paper_id, safe=':')}/references"
-    params: dict[str, Any] = {"fields": "externalIds,contexts"}
-    try:
-        response = _get(url, params=params)
-    except httpx.HTTPStatusError as exc:
-        if exc.response.status_code == 404:
-            return []
-        raise
-
-    data = response.json()
-    # S2 returns HTTP 200 with ``{"data": null}`` for a known paper that has no
-    # parsed reference list (e.g. some theses). ``.get("data", [])`` would yield
-    # the explicit ``None`` (the default only applies when the key is absent),
-    # so iterate over a guaranteed list instead.
-    raw_refs: list[dict[str, Any]] = data.get("data") or []
-    citations: list[Citation] = []
-    for entry in raw_refs:
-        cited_paper: dict[str, Any] = entry.get("citedPaper", {})
-        external_ids: dict[str, str] = cited_paper.get("externalIds") or {}
-        contexts: list[str] = entry.get("contexts", [])
-        context: str | None = contexts[0] if contexts else None
-
-        cited_id: str = (
-            external_ids.get("DOI") or external_ids.get("ArXiv") or cited_paper.get("paperId") or ""
-        )
-        if cited_id:
-            citations.append(Citation(citing_id=paper_id, cited_id=cited_id, context=context))
-    return citations
-
-
-def fetch_abstract(paper_id: str) -> str | None:
-    """Fetch just the abstract for a paper from Semantic Scholar.
-
-    Used as an ingest abstract supplement (DQ-2) when OpenAlex has the paper
-    but no abstract. ``paper_id`` should be namespaced (``arXiv:…`` / ``DOI:…``).
-    Returns None on 404 or when S2 has no abstract.
-    """
-    url = f"{_BASE_GRAPH}/paper/{paper_id}"
-    try:
-        response = _get(url, params={"fields": "abstract"})
-    except httpx.HTTPStatusError as exc:
-        if exc.response.status_code == 404:
-            return None
-        raise
-    abstract = response.json().get("abstract")
-    return abstract or None
-
-
-def fetch_recommendations(paper_id: str, limit: int = 20) -> list[str]:
-    """Fetch recommended paper IDs from Semantic Scholar.
-
-    Parameters
-    ----------
-    paper_id:
-        Semantic Scholar paper ID.
-    limit:
-        Maximum number of recommendations to return.
-
-    Returns
-    -------
-    list[str]
-        List of recommended paper IDs.
-    """
-    url = f"{_BASE_RECS}/papers/forpaper/{quote(paper_id, safe=':')}"
-    params: dict[str, Any] = {"limit": limit}
-    try:
-        response = _get(url, params=params)
-    except httpx.HTTPStatusError as exc:
-        if exc.response.status_code == 404:
-            return []
-        raise
-
-    data = response.json()
-    recommended: list[dict[str, Any]] = data.get("recommendedPapers", [])
-    return [p["paperId"] for p in recommended if p.get("paperId")]

codex/synthesis.py

@@ -1,874 +0,0 @@
-"""Synthesis / Lead-Engine (F-13).
-
-Generates four kinds of research **leads** over the RAG substrate:
-
-| Stage | Kind          | Grounded? | Output dir              |
-|-------|---------------|-----------|--------------------------|
-| 2     | connection    | yes       | ``leads/grounded/``     |
-| 3     | gap           | yes       | ``leads/grounded/``     |
-| 4     | improvement   | yes       | ``leads/grounded/``     |
-| 5     | conjecture    | no — by definition unverifiable | ``leads/conjectures/`` |
-
-Hard invariant (F-13 HARD)
---------------------------
-**Conjectures are NEVER written to ``wiki/``.** They are quarantined to
-``leads/conjectures/`` with mandatory ``status="unverified"`` plus a
-``suggested_validation`` field describing the spike that would falsify them.
-
-The wiki layer (F-12) stays grounded-only. Conjectures live in their own
-sandbox so that human review can promote them after a spike succeeds.
-
-Grounding discipline
---------------------
-* For ``connection`` / ``gap`` / ``improvement`` leads we run the
-  same content-5-gram grounding check as F-12 (re-used directly from
-  :mod:`codex.wiki`). Every claim in the lead body that fails the check
-  is marked with the ⚠ prefix and the lead is recorded as having
-  ungrounded claims. Leads whose grounded-ratio is below the floor are
-  dropped (returned as "ungrounded" rather than "grounded").
-* For ``conjecture`` leads grounding by definition cannot apply
-  (the claim does not exist in the corpus). Provenance is still mandatory:
-  it points at the chunks that *seeded* the conjecture so a reviewer can
-  trace the inspiration.
-
-Graceful degradation
---------------------
-* Missing ``lib_path`` → :func:`find_improvements` returns ``[]``.
-* DB unreachable → caller sees the psycopg error; no silent failures.
-* LLM unreachable (httpx.ConnectError / any exception) → the affected
-  generator returns ``[]`` (matches the F-12 LLM degradation rule).
-"""
-
-from __future__ import annotations
-
-import json
-import logging
-import re
-from dataclasses import asdict, dataclass, field
-from datetime import UTC, datetime
-from pathlib import Path
-from typing import Any, Literal, Protocol
-
-from codex.config import get_settings
-from codex.wiki import (
-    Claim,
-    LLMClient,
-    OllamaClient,
-    _parse_claims,
-    _retrieve_chunks,
-    _run_grounding_guard,
-)
-
-logger = logging.getLogger(__name__)
-
-# ---------------------------------------------------------------------------
-# Datamodel
-# ---------------------------------------------------------------------------
-
-LeadKind = Literal["connection", "gap", "improvement", "conjecture"]
-LeadStatus = Literal["unverified", "spiking", "go", "no-go"]
-
-
-@dataclass
-class Provenance:
-    """Pointer back to a single source location for a lead.
-
-    ``locator`` follows the same conventions as F-12 wiki claims:
-    ``"chunk 16"``, ``"page 9"``, ``"eq.(9)"``, or — for ``improvement``
-    leads — a code locator like ``"src/cli.cpp:42"``.
-    """
-
-    bibkey: str
-    locator: str
-
-
-@dataclass
-class Lead:
-    """A single research lead emitted by the synthesis engine.
-
-    See module docstring for the four-stage taxonomy and the
-    grounded-vs-conjecture invariant.
-    """
-
-    id: str  # "L-0001"
-    kind: LeadKind
-    title: str
-    body: str
-    provenance: list[Provenance]  # mandatory — even for conjectures
-    confidence: float  # 0..1
-    status: LeadStatus = "unverified"
-    # For conjectures: the falsification spike. Empty string for grounded leads.
-    suggested_validation: str = ""
-    # Derived: claim-level grounding decisions (empty for conjectures).
-    claims: list[Claim] = field(default_factory=list)
-
-    def __post_init__(self) -> None:
-        if not self.provenance:
-            raise ValueError(
-                f"Lead '{self.id}' must carry at least one Provenance entry. "
-                "For gap leads with no chunks, use a topic-marker Provenance."
-            )
-
-
-# ---------------------------------------------------------------------------
-# Protocols
-# ---------------------------------------------------------------------------
-
-
-class _ConnLike(Protocol):
-    """Subset of ``psycopg.Connection`` used by this module (helps tests)."""
-
-    def execute(self, sql: str, params: dict[str, Any] | None = ...) -> Any: ...
-
-
-# ---------------------------------------------------------------------------
-# LLM prompts
-# ---------------------------------------------------------------------------
-
-_CONNECTION_PROMPT = """\
-You are a research analyst surfacing CONNECTIONS between papers.
-
-A connection is a non-trivial overlap (shared definitions, shared techniques,
-shared theorems, or one paper extending/generalising another). It is a
-DESCRIPTIVE statement about the corpus — not a conjecture.
-
-Use ONLY the source chunks below. For every factual claim, cite it inline
-in the format [BibKey #locator]. Example:
-"Both papers use the Lobachevsky function as the building block of volume
-formulas. [Springborn2008 #chunk 16] [BobenkoPinkallSpringborn2015 #chunk 3]"
-
-Do NOT invent facts. Do NOT speculate. Two short paragraphs maximum.
-
-SOURCE CHUNKS:
-{chunks_block}
-
-Now describe a single non-trivial connection between these papers:
-"""
-
-_GAP_PROMPT = """\
-You are a research analyst identifying GAPS in a corpus.
-
-A gap is an ABSENCE: a topic, technique, or comparison that the corpus
-fails to cover, even though adjacent material exists. Make the absence
-concrete by pointing at where the corpus DOES talk about adjacent things.
-
-Use ONLY the source chunks below. For every factual claim about what IS
-in the corpus, cite it inline in the format [BibKey #locator].
-The absence itself does not need a citation; the surrounding context does.
-
-Do NOT invent facts. Two short paragraphs maximum.
-
-SOURCE CHUNKS:
-{chunks_block}
-
-Now describe a single concrete gap in the corpus's coverage of "{topic}":
-"""
-
-_IMPROVEMENT_PROMPT = """\
-You are a research analyst inspecting C++ code annotated with @cite tags.
-
-For the symbol below, a literature paper is cited. Read the chunks and
-suggest at most one CONCRETE improvement to the code that the literature
-would support. The improvement must be a DESCRIPTIVE statement of what the
-paper says, not a conjecture.
-
-Use ONLY the source chunks. Cite each claim inline as [BibKey #locator].
-
-Symbol:        {symbol}
-Cited bibkey:  {bibkey}
-File:locator:  {file_locator}
-
-SOURCE CHUNKS:
-{chunks_block}
-
-Now state a single grounded improvement (or "No improvement suggested." if
-the chunks do not support any):
-"""
-
-_CONJECTURE_PROMPT = """\
-You are a creative researcher proposing CONJECTURES — novel hypotheses that
-EXTEND but DO NOT contradict the corpus.
-
-A conjecture is a NEW idea not yet present in the literature. By definition
-it cannot be grounded in the chunks. Your output MUST be honest about this:
-do not pretend it is a result already in the papers.
-
-You MUST produce:
-1. A one-sentence title (the conjecture itself).
-2. A short body (2-4 sentences) explaining the intuition.
-3. A "Validation:" line stating ONE concrete spike — a small experiment,
-   counter-example search, or proof attempt — that would FALSIFY the
-   conjecture if it is wrong. Without a falsification path the conjecture
-   is rejected.
-
-Use the chunks below ONLY as inspiration. Cite seed material as
-[BibKey #locator] in the body so a reviewer can trace where the idea came
-from.
-
-SOURCE CHUNKS:
-{chunks_block}
-
-Now propose one conjecture in the format:
-Title: <one sentence>
-Body: <2-4 sentences with [BibKey #locator] seed citations>
-Validation: <one concrete falsification spike>
-"""
-
-
-# ---------------------------------------------------------------------------
-# Internal helpers
-# ---------------------------------------------------------------------------
-
-
-def _format_chunks(chunks: list[dict[str, Any]]) -> str:
-    """Render chunks for the LLM prompt — same format as F-12."""
-    lines: list[str] = []
-    for chunk in chunks:
-        bibkey = chunk.get("bibkey") or chunk["paper_id"]
-        ord_val = chunk.get("ord", "?")
-        lines.append(f"[{bibkey} #chunk {ord_val}]\n{chunk['content'].strip()}\n")
-    return "\n---\n".join(lines)
-
-
-def _mark_ungrounded(body: str, claims: list[Claim]) -> str:
-    """Prefix each ungrounded claim's last sentence with ⚠ in *body*."""
-    for claim in claims:
-        if claim.grounded:
-            continue
-        escaped = re.escape(claim.text.strip())
-        body = re.sub(rf"(?<!⚠ )({escaped})", r"⚠ \1", body, count=1)
-    return body
-
-
-def _grounded_ratio(claims: list[Claim]) -> float:
-    """Fraction of *claims* that passed the grounding check (0.0 if empty)."""
-    if not claims:
-        return 0.0
-    n_grounded = sum(1 for c in claims if c.grounded)
-    return n_grounded / len(claims)
-
-
-_KIND_PREFIX: dict[LeadKind, str] = {
-    "connection": "C",
-    "gap": "G",
-    "improvement": "I",
-    "conjecture": "X",
-}
-
-
-def _make_lead_id(seq: int, kind: LeadKind) -> str:
-    """Format a per-kind sequential lead id like ``L-C-0001``.
-
-    Each synthesis stage numbers its leads from 1, so without a kind prefix a
-    connection, a gap, and an improvement all produce ``L-0001`` and clobber
-    one another in the shared ``grounded/`` directory (audit C-11). The kind
-    prefix namespaces the ids so they stay distinct.
-    """
-    return f"L-{_KIND_PREFIX[kind]}-{seq:04d}"
-
-
-def _safe_llm_generate(llm: LLMClient, prompt: str, model: str) -> str:
-    """Call the LLM, return ``""`` on any exception (mirrors F-12 behaviour)."""
-    try:
-        return llm.generate(prompt, model=model)
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("LLM unavailable for synthesis (%s)", exc)
-        return ""
-
-
-def _provenance_from_chunks(chunks: list[dict[str, Any]]) -> list[Provenance]:
-    """Build Provenance pointers for every chunk used as a seed."""
-    out: list[Provenance] = []
-    for chunk in chunks:
-        bibkey = str(chunk.get("bibkey") or chunk.get("paper_id") or "")
-        ord_val = chunk.get("ord", "?")
-        out.append(Provenance(bibkey=bibkey, locator=f"chunk {ord_val}"))
-    return out
-
-
-def _finalise_grounded_lead(
-    *,
-    seq: int,
-    kind: LeadKind,
-    title: str,
-    body: str,
-    chunks: list[dict[str, Any]],
-    min_grounded_ratio: float,
-) -> Lead | None:
-    """Parse claims, run grounding guard, mark ⚠, return ``None`` if too sparse.
-
-    Returns the populated :class:`Lead` or ``None`` if no parseable claims
-    were found or the grounded ratio is below ``min_grounded_ratio``.
-    """
-    claims = _parse_claims(body)
-    if not claims:
-        # No citations at all → cannot be a grounded lead.
-        return None
-    claims = _run_grounding_guard(claims, chunks)
-    ratio = _grounded_ratio(claims)
-    if ratio < min_grounded_ratio:
-        return None
-    marked = _mark_ungrounded(body, claims)
-    return Lead(
-        id=_make_lead_id(seq, kind),
-        kind=kind,
-        title=title,
-        body=marked,
-        provenance=_provenance_from_chunks(chunks),
-        confidence=ratio,
-        status="unverified",
-        suggested_validation="",
-        claims=claims,
-    )
-
-
-# ---------------------------------------------------------------------------
-# Stage 2 — connections
-# ---------------------------------------------------------------------------
-
-
-def find_connections(
-    *,
-    db_conn: _ConnLike | None = None,
-    top_k: int,
-    llm: LLMClient,
-) -> list[Lead]:
-    """Surface grounded connection leads across the corpus.
-
-    Strategy: cluster the corpus by shared retrieval — for each paper
-    pair that shows up together in the top-K of a common probe query
-    we ask the LLM to articulate the connection. The probe queries are
-    the bibkey/title pairs of the existing papers (cheap, no external
-    config needed).
-
-    Parameters
-    ----------
-    db_conn:
-        Unused right now (kept for the agreed signature so callers can
-        inject a transactional connection later).
-    top_k:
-        Number of chunks retrieved per probe.
-    llm:
-        Injectable LLM client. Errors fall back to ``[]``.
-    """
-    settings = get_settings()
-    seq = 1
-    leads: list[Lead] = []
-
-    # Probe queries: use the discovered titles of already-ingested papers.
-    # Without DB we can't get titles, so we fall back to a generic probe.
-    titles = _list_paper_titles(db_conn)
-    if not titles:
-        return leads
-
-    seen_pairs: set[tuple[str, str]] = set()
-
-    for bibkey, title in titles:
-        chunks = _retrieve_chunks([title], top_k=top_k)
-        if not chunks:
-            continue
-        # Find chunks from a *different* bibkey — that's the connection signal.
-        other_bibkeys = {
-            str(c.get("bibkey") or "")
-            for c in chunks
-            if c.get("bibkey") and c.get("bibkey") != bibkey
-        }
-        for other in sorted(other_bibkeys):
-            pair = tuple(sorted([bibkey, other]))
-            if pair in seen_pairs:
-                continue
-            seen_pairs.add(pair)  # type: ignore[arg-type]
-
-            pair_chunks = [c for c in chunks if c.get("bibkey") in {bibkey, other}]
-            prompt = _CONNECTION_PROMPT.format(chunks_block=_format_chunks(pair_chunks))
-            raw = _safe_llm_generate(llm, prompt, settings.synthesis_llm_model)
-            if not raw.strip():
-                continue
-            lead = _finalise_grounded_lead(
-                seq=seq,
-                kind="connection",
-                title=f"Connection: {bibkey} ⇄ {other}",
-                body=raw.strip(),
-                chunks=pair_chunks,
-                min_grounded_ratio=settings.synthesis_min_grounded_ratio,
-            )
-            if lead is None:
-                continue
-            leads.append(lead)
-            seq += 1
-
-    return leads
-
-
-# ---------------------------------------------------------------------------
-# Stage 3 — gaps
-# ---------------------------------------------------------------------------
-
-
-def find_gaps(
-    lib_path: str | None = None,
-    *,
-    db_conn: _ConnLike | None = None,
-    llm: LLMClient,
-    topics: list[str] | None = None,
-) -> list[Lead]:
-    """Surface grounded gap leads.
-
-    Two complementary signals are used:
-
-    * **Topic coverage map** — for each **explicitly-requested** topic in
-      ``topics``, retrieve chunks and check bibkey coverage. A topic covered by
-      < ``synthesis_gap_min_coverage`` bibkeys is a gap candidate and the LLM is
-      asked to articulate it. Defaulting to paper titles flagged almost every
-      paper as a gap (a title retrieves mostly its own single bibkey), so the
-      coverage map is opt-in now; pass ``topics`` / CLI ``--topic`` (audit R-8).
-    * **Code-vs-corpus gap** — when ``lib_path`` is given, any @cite
-      bibkey scanned from code that is NOT present in the corpus is
-      flagged. This catches the "cited in code, never ingested" case.
-    """
-    settings = get_settings()
-    seq = 1
-    leads: list[Lead] = []
-
-    # Coverage-map probes are the explicit topics only — no default-to-titles,
-    # which turned every paper into a spurious gap (audit R-8).
-    probe_topics: list[str] = list(topics or [])
-
-    known_bibkeys = {bk for bk, _ in _list_paper_titles(db_conn)}
-
-    # --- coverage-map gaps --------------------------------------------
-    for topic in probe_topics:
-        chunks = _retrieve_chunks([topic], top_k=settings.synthesis_top_k)
-        if not chunks:
-            # No coverage at all → record the gap directly (LLM not strictly needed).
-            leads.append(
-                Lead(
-                    id=_make_lead_id(seq, "gap"),
-                    kind="gap",
-                    title=f"Gap: corpus has no material on '{topic}'",
-                    body=(
-                        f"The retrieval layer returned no chunks for topic '{topic}'.\n"
-                        "No grounded statement is possible; this is the absence itself."
-                    ),
-                    provenance=[Provenance(bibkey="(none)", locator=f"topic:{topic}")],
-                    confidence=1.0,
-                    status="unverified",
-                    suggested_validation=(
-                        "Ingest a paper covering this topic, re-run synthesis: the gap "
-                        "should disappear."
-                    ),
-                )
-            )
-            seq += 1
-            continue
-
-        bibkeys_for_topic = {str(c.get("bibkey") or "") for c in chunks if c.get("bibkey")}
-        if len(bibkeys_for_topic) >= settings.synthesis_gap_min_coverage:
-            # Topic is well covered — not a gap.
-            continue
-        prompt = _GAP_PROMPT.format(topic=topic, chunks_block=_format_chunks(chunks))
-        raw = _safe_llm_generate(llm, prompt, settings.synthesis_llm_model)
-        if not raw.strip():
-            continue
-        lead = _finalise_grounded_lead(
-            seq=seq,
-            kind="gap",
-            title=f"Gap: '{topic}' covered by only {len(bibkeys_for_topic)} bibkey(s)",
-            body=raw.strip(),
-            chunks=chunks,
-            min_grounded_ratio=settings.synthesis_min_grounded_ratio,
-        )
-        if lead is None:
-            continue
-        leads.append(lead)
-        seq += 1
-
-    # --- code-vs-corpus gaps ------------------------------------------
-    if lib_path:
-        from codex.provenance import scan_cite_tags
-
-        try:
-            hits = scan_cite_tags(lib_path)
-        except FileNotFoundError:
-            hits = []
-        cited_bibkeys = {h["bibkey"] for h in hits}
-        missing = cited_bibkeys - known_bibkeys
-        for bibkey in sorted(missing):
-            leads.append(
-                Lead(
-                    id=_make_lead_id(seq, "gap"),
-                    kind="gap",
-                    title=f"Gap: code cites '{bibkey}' but corpus does not contain it",
-                    body=(
-                        f"The bibkey `{bibkey}` is referenced by @cite tags in `{lib_path}`\n"
-                        "but no paper with this bibkey is ingested into the corpus.\n"
-                        "This is a concrete absence."
-                    ),
-                    provenance=[Provenance(bibkey=bibkey, locator="(not ingested)")],
-                    confidence=1.0,
-                    status="unverified",
-                    suggested_validation=(
-                        f"Ingest the paper for `{bibkey}` via `codex ingest`. The gap "
-                        "should disappear on the next synthesis run."
-                    ),
-                )
-            )
-            seq += 1
-
-    return leads
-
-
-# ---------------------------------------------------------------------------
-# Stage 4 — improvements
-# ---------------------------------------------------------------------------
-
-
-def find_improvements(
-    lib_path: str,
-    *,
-    db_conn: _ConnLike | None = None,
-    top_k: int,
-    llm: LLMClient,
-) -> list[Lead]:
-    """Suggest grounded code improvements based on @cite-linked literature.
-
-    Walks ``lib_path`` for ``@cite <bibkey>`` markers, retrieves the
-    relevant chunks for each cited bibkey, and asks the LLM to propose
-    one concrete improvement supported by the paper. Improvements
-    failing the grounding check are dropped.
-    """
-    if not lib_path:
-        return []
-    from codex.provenance import scan_cite_tags
-
-    settings = get_settings()
-    seq = 1
-    leads: list[Lead] = []
-
-    try:
-        hits = scan_cite_tags(lib_path)
-    except FileNotFoundError:
-        return leads
-
-    known_bibkeys = {bk for bk, _ in _list_paper_titles(db_conn)}
-
-    # Group hits by (symbol, bibkey) so we ask the LLM at most once per pair.
-    by_key: dict[tuple[str, str], list[dict[str, str]]] = {}
-    for hit in hits:
-        if hit["bibkey"] not in known_bibkeys:
-            continue  # handled by find_gaps
-        by_key.setdefault((hit["symbol"] or "(unknown)", hit["bibkey"]), []).append(hit)
-
-    for (symbol, bibkey), group in by_key.items():
-        chunks = _retrieve_chunks([symbol, bibkey], top_k=top_k)
-        # Restrict to chunks from the cited bibkey when possible
-        same_bib = [c for c in chunks if c.get("bibkey") == bibkey] or chunks
-        if not same_bib:
-            continue
-        sample = group[0]
-        file_locator = f"{sample['file']}:{sample['line']}"
-        prompt = _IMPROVEMENT_PROMPT.format(
-            symbol=symbol,
-            bibkey=bibkey,
-            file_locator=file_locator,
-            chunks_block=_format_chunks(same_bib),
-        )
-        raw = _safe_llm_generate(llm, prompt, settings.synthesis_llm_model)
-        if not raw.strip() or "no improvement suggested" in raw.lower():
-            continue
-        lead = _finalise_grounded_lead(
-            seq=seq,
-            kind="improvement",
-            title=f"Improvement: {symbol} ({file_locator}) via {bibkey}",
-            body=raw.strip(),
-            chunks=same_bib,
-            min_grounded_ratio=settings.synthesis_min_grounded_ratio,
-        )
-        if lead is None:
-            continue
-        # Attach the code locator to the provenance list as the first entry
-        lead.provenance.insert(0, Provenance(bibkey=bibkey, locator=file_locator))
-        leads.append(lead)
-        seq += 1
-
-    return leads
-
-
-# ---------------------------------------------------------------------------
-# Stage 5 — conjectures
-# ---------------------------------------------------------------------------
-
-
-_CONJECTURE_RE = re.compile(
-    r"Title:\s*(?P<title>.+?)\n+"
-    r"Body:\s*(?P<body>.+?)\n+"
-    r"Validation:\s*(?P<validation>.+)",
-    re.DOTALL | re.IGNORECASE,
-)
-
-
-def _parse_conjecture_output(raw: str) -> tuple[str, str, str] | None:
-    """Parse the LLM's three-section conjecture output.
-
-    Returns ``(title, body, validation)`` or ``None`` if any section
-    is missing or empty.
-    """
-    match = _CONJECTURE_RE.search(raw)
-    if not match:
-        return None
-    title = match.group("title").strip()
-    body = match.group("body").strip()
-    validation = match.group("validation").strip()
-    if not title or not body or not validation:
-        return None
-    return title, body, validation
-
-
-def propose_conjectures(
-    *,
-    db_conn: _ConnLike | None = None,
-    top_k: int,
-    llm: LLMClient,
-    seeds: list[str] | None = None,
-) -> list[Lead]:
-    """Generate stage-5 conjecture leads.
-
-    Each conjecture is the LLM's own hypothesis seeded by a small
-    retrieval slice. The output is by definition ungrounded, so:
-
-    * ``status`` is hard-set to ``"unverified"``.
-    * ``suggested_validation`` is mandatory; conjectures without a
-      falsification path are dropped.
-    * ``provenance`` records the seed chunks so a reviewer can trace
-      inspiration.
-
-    Conjectures are written to ``leads/conjectures/`` only — never
-    to ``wiki/``.
-    """
-    settings = get_settings()
-    seq = 1
-    leads: list[Lead] = []
-
-    probe_seeds: list[str] = list(seeds or [])
-    if not probe_seeds:
-        probe_seeds = [title for _, title in _list_paper_titles(db_conn)]
-    if not probe_seeds:
-        return leads
-
-    for seed in probe_seeds:
-        chunks = _retrieve_chunks([seed], top_k=top_k)
-        if not chunks:
-            continue
-        prompt = _CONJECTURE_PROMPT.format(chunks_block=_format_chunks(chunks))
-        raw = _safe_llm_generate(llm, prompt, settings.synthesis_llm_model)
-        if not raw.strip():
-            continue
-        parsed = _parse_conjecture_output(raw)
-        if parsed is None:
-            # No falsification path → reject (hard invariant).
-            continue
-        title, body, validation = parsed
-        provenance = _provenance_from_chunks(chunks)
-        if not provenance:
-            continue
-        leads.append(
-            Lead(
-                id=_make_lead_id(seq, "conjecture"),
-                kind="conjecture",
-                title=title,
-                body=body,
-                provenance=provenance,
-                confidence=0.0,  # conjectures start unconfirmed
-                status="unverified",  # HARD invariant
-                suggested_validation=validation,
-            )
-        )
-        seq += 1
-    return leads
-
-
-# ---------------------------------------------------------------------------
-# DB helpers
-# ---------------------------------------------------------------------------
-
-
-def _list_paper_titles(db_conn: _ConnLike | None) -> list[tuple[str, str]]:
-    """Return ``[(bibkey, title), …]`` for all ingested papers with a bibkey.
-
-    Uses the injected connection when provided, otherwise opens a fresh
-    one. Returns ``[]`` if the database is unreachable or has no rows.
-    """
-    sql = "SELECT bibkey, title FROM papers WHERE bibkey IS NOT NULL"
-    try:
-        if db_conn is not None:
-            rows = db_conn.execute(sql).fetchall()
-        else:
-            from codex.db import get_conn
-
-            with get_conn() as conn:
-                rows = conn.execute(sql).fetchall()
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("Could not list paper titles (%s)", exc)
-        return []
-    out: list[tuple[str, str]] = []
-    for row in rows:
-        bibkey = str(row["bibkey"]) if "bibkey" in row else str(row[0])
-        title = str(row["title"]) if "title" in row else str(row[1])
-        out.append((bibkey, title))
-    return out
-
-
-# ---------------------------------------------------------------------------
-# Persistence
-# ---------------------------------------------------------------------------
-
-
-_GROUNDED_KINDS: frozenset[LeadKind] = frozenset({"connection", "gap", "improvement"})
-
-
-def _lead_to_markdown(lead: Lead) -> str:
-    """Render a Lead as a self-contained markdown document.
-
-    Front-matter is a fenced JSON block so the file is human-readable
-    and machine-parseable.
-    """
-    meta = {
-        "id": lead.id,
-        "kind": lead.kind,
-        "status": lead.status,
-        "confidence": round(lead.confidence, 3),
-        "provenance": [asdict(p) for p in lead.provenance],
-        "suggested_validation": lead.suggested_validation,
-        "compiled_at": datetime.now(UTC).isoformat(),
-    }
-    parts = [
-        f"# {lead.id} — {lead.title}",
-        "",
-        "```json",
-        json.dumps(meta, indent=2, sort_keys=True),
-        "```",
-        "",
-        lead.body.rstrip(),
-        "",
-    ]
-    if lead.kind == "conjecture":
-        parts.extend(
-            [
-                "## Status",
-                "",
-                "**UNVERIFIED CONJECTURE — not grounded in the corpus.**",
-                "Promotion to the wiki requires the validation spike below to succeed.",
-                "",
-                "## Suggested validation",
-                "",
-                lead.suggested_validation,
-                "",
-            ]
-        )
-    return "\n".join(parts)
-
-
-def write_leads(leads: list[Lead], leads_dir: str) -> None:
-    """Write ``leads`` to disk under ``leads_dir``.
-
-    * Grounded leads → ``leads_dir/grounded/<id>.md``
-    * Conjectures   → ``leads_dir/conjectures/<id>.md`` (HARD invariant)
-
-    The function ALSO rebuilds ``leads_dir/INDEX.md`` from the on-disk files on
-    every run (see :func:`_update_index`): it reflects whatever lead files
-    currently exist under ``grounded/`` and ``conjectures/``, listed as two
-    visibly separated sections. It is not append-only — a deleted lead file
-    drops out of the index on the next run.
-
-    Notes
-    -----
-    * The function asserts the kind→directory routing — a conjecture
-      ending up under ``grounded/`` would be an invariant violation
-      and is raised as :class:`RuntimeError`.
-    """
-    root = Path(leads_dir)
-    grounded_dir = root / "grounded"
-    conj_dir = root / "conjectures"
-    grounded_dir.mkdir(parents=True, exist_ok=True)
-    conj_dir.mkdir(parents=True, exist_ok=True)
-
-    for lead in leads:
-        markdown = _lead_to_markdown(lead)
-        if lead.kind == "conjecture":
-            target_dir = conj_dir
-        elif lead.kind in _GROUNDED_KINDS:
-            target_dir = grounded_dir
-        else:  # pragma: no cover — defensive
-            raise RuntimeError(f"Unknown lead kind: {lead.kind!r}")
-
-        # HARD invariant: conjectures NEVER land in grounded/ or wiki/.
-        if lead.kind == "conjecture" and target_dir != conj_dir:
-            raise RuntimeError(
-                f"INVARIANT VIOLATION: conjecture {lead.id} would be written "
-                f"to {target_dir} instead of {conj_dir}"
-            )
-
-        path = target_dir / f"{lead.id}.md"
-        path.write_text(markdown, encoding="utf-8")
-
-    # Update INDEX.md
-    _update_index(root)
-
-
-def _update_index(root: Path) -> None:
-    """Re-build ``root/INDEX.md`` from the on-disk files (full rebuild, not append-only)."""
-    grounded_dir = root / "grounded"
-    conj_dir = root / "conjectures"
-    grounded_files = sorted(grounded_dir.glob("*.md")) if grounded_dir.exists() else []
-    conj_files = sorted(conj_dir.glob("*.md")) if conj_dir.exists() else []
-
-    ts = datetime.now(UTC).strftime("%Y-%m-%d %H:%M UTC")
-    lines = [
-        "# Lead Index",
-        "",
-        f"_Last updated {ts} by `codex synthesis`_",
-        "",
-        "## Grounded leads (connection / gap / improvement)",
-        "",
-    ]
-    if grounded_files:
-        for f in grounded_files:
-            title = _extract_title(f) or f.stem
-            lines.append(f"- `{f.stem}` — {title}")
-    else:
-        lines.append("_None._")
-    lines.extend(["", "## Conjectures (UNVERIFIED — quarantined here, never in wiki/)", ""])
-    if conj_files:
-        for f in conj_files:
-            title = _extract_title(f) or f.stem
-            lines.append(f"- `{f.stem}` — {title}")
-    else:
-        lines.append("_None._")
-    lines.append("")
-    (root / "INDEX.md").write_text("\n".join(lines), encoding="utf-8")
-
-
-def _extract_title(path: Path) -> str | None:
-    """Pull the H1 title from a lead markdown file, if present."""
-    try:
-        first = path.read_text(encoding="utf-8").splitlines()[0]
-    except (OSError, IndexError):
-        return None
-    if first.startswith("# "):
-        return first[2:].strip()
-    return None
-
-
-# ---------------------------------------------------------------------------
-# Default LLM client
-# ---------------------------------------------------------------------------
-
-
-def default_llm_client() -> LLMClient:
-    """Return an :class:`OllamaClient` configured from the synthesis settings.
-
-    Falls back to ``ollama_base_url`` when ``synthesis_llm_url`` is unset.
-    """
-    settings = get_settings()
-    url = settings.synthesis_llm_url or settings.ollama_base_url
-    return OllamaClient(url)

codex/wiki.py

@@ -1,883 +0,0 @@
-"""Wiki-compile layer — grounded concept pages over the RAG substrate (F-12).
-
-Each concept page is compiled from retrieved chunks via a local LLM (Ollama).
-Every claim is grounded against its cited source chunk (Substring-Match MVP).
-Cross-references to other concepts are rendered as ``[[slug]]`` links.
-Generated pages are written to ``wiki/<slug>.md`` and committed to git.
-
-Compile-state (hash tracking for incremental re-runs) is persisted to
-``wiki/.compile-state.json`` on disk — no DB changes (F-12 constraint).
-
-Graceful degradation:
-- Missing ``formulas`` table (F-09 not present) → no formula embedding, no crash.
-- Missing ``verify_citations`` (F-10 not present) → local substring grounding check.
-- LLM unavailable (httpx.ConnectError or any exception) → empty ConceptPage, no crash.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import logging
-import re
-import textwrap
-from dataclasses import dataclass, field
-from datetime import UTC, datetime
-from pathlib import Path
-from typing import Any, Protocol
-
-import yaml
-
-from codex.config import get_settings
-
-logger = logging.getLogger(__name__)
-
-# ---------------------------------------------------------------------------
-# Dataclasses
-# ---------------------------------------------------------------------------
-
-
-@dataclass
-class Concept:
-    """A curated concept seed from ``wiki/concepts.yaml``."""
-
-    slug: str
-    title: str
-    aliases: list[str]
-    emphasis: str | None = None
-
-
-@dataclass
-class Claim:
-    """A single factual claim extracted from a synthesised concept page."""
-
-    text: str
-    bibkey: str
-    locator: str  # e.g. "page 9" | "eq.(9)" | "chunk 42"
-    grounded: bool = True  # set by Grounding-Guard
-
-
-@dataclass
-class ConceptPage:
-    """The compiled wiki page for one concept."""
-
-    concept: Concept
-    markdown: str  # final rendered markdown
-    claims: list[Claim] = field(default_factory=list)
-    chunk_hash: str = ""  # SHA-256 of concatenated source chunks
-    compiled_at: datetime = field(default_factory=lambda: datetime.now(UTC))
-
-
-@dataclass
-class CompileReport:
-    """Summary of a compile run (appended to ``wiki/log.md``)."""
-
-    ran_at: datetime = field(default_factory=lambda: datetime.now(UTC))
-    compiled: list[str] = field(default_factory=list)  # slugs written/updated
-    skipped: list[str] = field(default_factory=list)  # slugs skipped (unchanged)
-    ungrounded: list[tuple[str, str]] = field(default_factory=list)  # (slug, claim_text)
-    conflicts: list[tuple[str, str, str]] = field(default_factory=list)  # (slug, bibkey1, bibkey2)
-    quarantined: list[str] = field(default_factory=list)  # slugs written to wiki/draft/
-
-
-# ---------------------------------------------------------------------------
-# LLM protocol (injectable for tests)
-# ---------------------------------------------------------------------------
-
-
-class LLMClient(Protocol):
-    """Minimal protocol for an LLM that can generate text."""
-
-    def generate(self, prompt: str, model: str) -> str:
-        """Return generated text for *prompt* using *model*."""
-        ...
-
-
-# ---------------------------------------------------------------------------
-# Default Ollama LLM client
-# ---------------------------------------------------------------------------
-
-
-class OllamaClient:
-    """Thin HTTP wrapper around the Ollama ``/api/generate`` endpoint."""
-
-    def __init__(self, base_url: str) -> None:
-        self._base_url = base_url.rstrip("/")
-
-    def generate(self, prompt: str, model: str) -> str:  # noqa: D102
-        import httpx
-
-        url = f"{self._base_url}/api/generate"
-        payload = {"model": model, "prompt": prompt, "stream": False}
-        response = httpx.post(url, json=payload, timeout=120.0)
-        response.raise_for_status()
-        data: dict[str, Any] = response.json()
-        return str(data.get("response", ""))
-
-
-# ---------------------------------------------------------------------------
-# YAML loader
-# ---------------------------------------------------------------------------
-
-
-def load_concepts(path: str) -> list[Concept]:
-    """Parse ``wiki/concepts.yaml`` and return a list of :class:`Concept` objects.
-
-    Each entry must have ``slug``, ``title``, and ``aliases`` (list).
-    ``emphasis`` is optional.
-    """
-    raw = Path(path).read_text(encoding="utf-8")
-    data: dict[str, Any] = yaml.safe_load(raw)
-    concepts: list[Concept] = []
-    for entry in data.get("concepts", []):
-        concepts.append(
-            Concept(
-                slug=str(entry["slug"]),
-                title=str(entry["title"]),
-                aliases=[str(a) for a in entry.get("aliases", [])],
-                emphasis=entry.get("emphasis") or None,
-            )
-        )
-    return concepts
-
-
-# ---------------------------------------------------------------------------
-# Retrieval helpers
-# ---------------------------------------------------------------------------
-
-
-def _retrieve_chunks(
-    queries: list[str],
-    *,
-    top_k: int,
-) -> list[dict[str, Any]]:
-    """Retrieve chunks via hybrid search (dense + FTS) from the DB.
-
-    Returns a list of dicts with keys: ``id``, ``paper_id``, ``ord``,
-    ``content``, ``bibkey``.  Reference-list chunks are filtered out
-    (ADR-F12: bibliography fragments pollute top-K).
-    """
-    from codex.db import get_conn
-    from codex.embed import get_embedder
-
-    embedder = get_embedder()
-    combined_query = " ".join(queries)
-    dense_vec = embedder.encode_dense([combined_query])[0].tolist()
-
-    sql = """
-        SELECT
-            c.id,
-            c.paper_id,
-            c.ord,
-            c.content,
-            p.bibkey,
-            c.embedding <-> %(emb)s::vector AS dist
-        FROM chunks c
-        JOIN papers p ON p.id = c.paper_id
-        WHERE c.embedding IS NOT NULL
-          AND p.bibkey IS NOT NULL
-        ORDER BY c.embedding <-> %(emb)s::vector
-        LIMIT %(top_k)s
-    """
-    with get_conn() as conn:
-        rows = conn.execute(
-            sql,
-            {"emb": dense_vec, "top_k": top_k * 2},  # over-fetch before filtering
-        ).fetchall()
-
-    # Filter reference-list chunks: skip chunks whose content looks like a bibliography
-    # (heuristic: > 60 % of lines look like "[12]" refs or "Surname, I." authors).
-    # The author alternative excludes common math-structure words ("Theorem A.",
-    # "Lemma B.") so theorem-dense chunks aren't mistaken for a reference list (R-2).
-    ref_pattern = re.compile(
-        r"^\s*(\[\d+\]|(?!(?:Theorem|Lemma|Proposition|Corollary|Definition|Proof|"
-        r"Section|Figure|Fig|Table|Equation|Eq|Remark|Example|Chapter)\b)"
-        r"[A-Z][a-z]+,?\s+[A-Z]\.)",
-        re.MULTILINE,
-    )
-    filtered: list[dict[str, Any]] = []
-    for row in rows:
-        content: str = row["content"]
-        lines = content.splitlines()
-        if not lines:
-            continue
-        ref_hits = len(ref_pattern.findall(content))
-        if ref_hits / max(len(lines), 1) > 0.6:
-            continue  # skip reference-list chunk
-        filtered.append(dict(row))
-        if len(filtered) >= top_k:
-            break
-
-    return filtered
-
-
-# ---------------------------------------------------------------------------
-# Grounding guard
-# ---------------------------------------------------------------------------
-
-_CLAIM_RE = re.compile(
-    r"(?P<text>[^\[]+?)\s*\[(?P<bibkey>[^\],#]+)(?:#(?P<locator>[^\]]+))?\]",
-)
-
-
-def _parse_claims(markdown: str) -> list[Claim]:
-    """Extract inline citations from the LLM output.
-
-    Expected format per claim::
-
-        Some factual statement. [BibKey2008 #page 9]
-
-    Returns a :class:`Claim` with ``text``, ``bibkey``, ``locator``.
-    The ``grounded`` flag defaults to ``True`` and is set by
-    :func:`_run_grounding_guard`.
-    """
-    claims: list[Claim] = []
-    for match in _CLAIM_RE.finditer(markdown):
-        text = match.group("text").strip()
-        bibkey = match.group("bibkey").strip()
-        locator = (match.group("locator") or "").strip()
-        # Skip URL-shaped bibkeys ([text](https://...)) and multi-word bibkeys
-        # (real BibKeys never contain spaces or start with "http")
-        if " " in bibkey or bibkey.startswith("http"):
-            continue
-        if text and bibkey:
-            claims.append(Claim(text=text, bibkey=bibkey, locator=locator))
-    return claims
-
-
-_STOPWORDS: frozenset[str] = frozenset(
-    {
-        "the",
-        "a",
-        "an",
-        "in",
-        "on",
-        "of",
-        "to",
-        "is",
-        "are",
-        "was",
-        "were",
-        "and",
-        "or",
-        "but",
-        "for",
-        "with",
-        "this",
-        "that",
-        "it",
-        "we",
-        "they",
-        "be",
-        "as",
-        "at",
-        "by",
-        "from",
-        "has",
-        "have",
-        "not",
-        "which",
-    }
-)
-
-# Split on sentence-ending punctuation only when followed by whitespace/end, so a
-# decimal like "1.5" does not create a spurious sentence boundary (audit R-1).
-_SENTENCE_SPLIT_RE = re.compile(r"[.!?]+(?=\s|$)")
-
-# Punctuation stripped from token edges before grounding comparison — sentence/
-# clause marks and quotes, NOT math delimiters like ()=+ (audit R-1).
-_EDGE_PUNCT = ".,;:!?\"'"
-
-
-def _last_sentence(text: str) -> str:
-    """Return the last non-empty sentence from *text*.
-
-    Citations annotate the last sentence, not the whole preceding paragraph.
-    Splitting on ``.``, ``!``, ``?`` and taking the last non-empty element
-    ensures only the immediately preceding sentence is grounding-checked.
-    """
-    parts = _SENTENCE_SPLIT_RE.split(text)
-    for part in reversed(parts):
-        stripped = part.strip()
-        if stripped:
-            return stripped
-    return text.strip()
-
-
-def _content_words(text: str) -> list[str]:
-    """Return lowercased content tokens, edge-punctuation-stripped, stopwords removed.
-
-    Stripping leading/trailing sentence punctuation makes grounding robust to
-    surface variation ("volume," vs "volume") so a faithful paraphrase is not
-    marked ungrounded over a comma (audit R-1). Math delimiters (``()=+``) are
-    preserved. Claim and chunk tokens are normalised identically, so matching
-    stays consistent.
-    """
-    tokens = (w.strip(_EDGE_PUNCT) for w in text.lower().split())
-    return [w for w in tokens if w and w not in _STOPWORDS]
-
-
-def _run_grounding_guard(
-    claims: list[Claim],
-    chunks: list[dict[str, Any]],
-) -> list[Claim]:
-    """Check each claim against its cited chunk via content-5-gram match.
-
-    A claim is *grounded* if the **last sentence** before its citation
-    contains at least one sequence of 5 consecutive non-stopword tokens
-    (a "content-5-gram") that also appears as 5 consecutive non-stopword
-    tokens in any chunk attributed to the same bibkey.
-
-    Matching is done in content-word space (stopwords removed from BOTH
-    claim and chunk), which prevents trivial bypass via common scientific
-    phrases like "the discrete conformal map" (only 2 content words).
-
-    Using sentence-level granularity prevents a hallucinated paragraph from
-    being grounded by a single matching phrase at its end.
-
-    Sets ``claim.grounded = False`` for any claim that fails this check.
-    """
-    # Build a bibkey → [content-word sequence] index
-    # We store the content-word list (joined as string for fast substring search)
-    bib_index: dict[str, list[str]] = {}
-    for chunk in chunks:
-        bk = str(chunk.get("bibkey") or "")
-        if bk:
-            # Content words of the chunk, space-padded so a 5-gram match is bounded
-            # by whole tokens, not a substring bleeding across words (audit C-6).
-            cw = " " + " ".join(_content_words(chunk["content"])) + " "
-            bib_index.setdefault(bk, []).append(cw)
-
-    for claim in claims:
-        sources = bib_index.get(claim.bibkey)
-        if not sources:
-            claim.grounded = False
-            continue
-
-        # Grounding operates on the LAST SENTENCE only
-        sentence = _last_sentence(claim.text)
-        content = _content_words(sentence)
-
-        if len(content) < 5:
-            # Too short for a content-5-gram → cannot be grounded.
-            # A claim sentence with fewer than 5 non-stopword tokens provides
-            # insufficient signal and is marked ungrounded by default.
-            claim.grounded = False
-            continue
-
-        found = False
-        for i in range(len(content) - 4):  # content-5-grams
-            gram = " " + " ".join(content[i : i + 5]) + " "  # padded: token-boundary match
-            if any(gram in src for src in sources):
-                found = True
-                break
-        claim.grounded = found
-
-    return claims
-
-
-# ---------------------------------------------------------------------------
-# Conflict detection (MVP: keyword-based signal)
-# ---------------------------------------------------------------------------
-
-_CONFLICT_KEYWORDS = re.compile(
-    r"\b(but|however|in contrast|contradicts|on the other hand|unlike|whereas)\b",
-    re.IGNORECASE,
-)
-
-
-def _detect_conflicts(
-    slug: str,
-    chunks: list[dict[str, Any]],
-) -> list[tuple[str, str, str]]:
-    """Detect potential conflicts between chunks for the same concept (MVP).
-
-    Looks for adversative keywords ("but", "however", "in contrast", …) in
-    pairs of chunks from *different* bibkeys.  Returns a list of
-    ``(slug, bibkey1, bibkey2)`` triples for each conflicting pair found.
-    """
-    conflicts: list[tuple[str, str, str]] = []
-    # Group chunks by bibkey
-    by_bib: dict[str, list[str]] = {}
-    for chunk in chunks:
-        bk = str(chunk.get("bibkey") or "")
-        if bk:
-            by_bib.setdefault(bk, []).append(chunk["content"])
-
-    bibkeys = list(by_bib.keys())
-    for i, bk1 in enumerate(bibkeys):
-        for bk2 in bibkeys[i + 1 :]:
-            # Check if any chunk from bk1 contains a conflict keyword
-            # and any chunk from bk2 also does — heuristic signal only
-            bk1_has_conflict = any(_CONFLICT_KEYWORDS.search(c) for c in by_bib[bk1])
-            bk2_has_conflict = any(_CONFLICT_KEYWORDS.search(c) for c in by_bib[bk2])
-            if bk1_has_conflict and bk2_has_conflict:
-                conflicts.append((slug, bk1, bk2))
-
-    return conflicts
-
-
-# ---------------------------------------------------------------------------
-# Cross-reference injection
-# ---------------------------------------------------------------------------
-
-
-_INLINE_CODE_RE = re.compile(r"`[^`]+`")
-
-# LaTeX math spans — protected from cross-ref injection so a concept name inside
-# a formula is not rewritten to a [[slug]] link, corrupting the LaTeX (audit R-9).
-_MATH_SPAN_RE = re.compile(
-    r"\$\$.*?\$\$|\$[^$\n]*?\$|\\\(.*?\\\)|\\\[.*?\\\]",
-    re.DOTALL,
-)
-
-
-def _inject_cross_refs(
-    markdown: str,
-    all_concepts: list[Concept],
-    current_slug: str,
-) -> str:
-    """Replace occurrences of other concept titles/aliases with ``[[slug]]`` links.
-
-    Only exact case-insensitive whole-word matches outside of existing
-    ``[[…]]`` blocks, inline code spans, or LaTeX math spans are replaced.
-    Inline-code (`` `…` ``) and math (``$…$``, ``$$…$$``, ``\\(…\\)``,
-    ``\\[…\\]``) spans are temporarily protected by null-byte placeholders and
-    restored after injection.
-    """
-    # Step 1: protect inline-code and math spans from replacement
-    placeholders: dict[str, str] = {}
-
-    def _protect(m: re.Match[str]) -> str:
-        key = f"\x00{len(placeholders)}\x00"
-        placeholders[key] = m.group(0)
-        return key
-
-    markdown = _INLINE_CODE_RE.sub(_protect, markdown)
-    markdown = _MATH_SPAN_RE.sub(_protect, markdown)  # don't rewrite names inside LaTeX (R-9)
-
-    # Step 2: inject cross-refs on unprotected text
-    for concept in all_concepts:
-        if concept.slug == current_slug:
-            continue
-        terms = [concept.title] + concept.aliases
-        for term in terms:
-            # Escape for use in regex; require word boundary
-            escaped = re.escape(term)
-            pattern = re.compile(rf"(?<!\[\[)\b{escaped}\b(?!\]\])", re.IGNORECASE)
-            replacement = f"[[{concept.slug}]]"
-            markdown = pattern.sub(replacement, markdown)
-
-    # Step 3: restore inline-code spans
-    for key, val in placeholders.items():
-        markdown = markdown.replace(key, val)
-
-    return markdown
-
-
-# ---------------------------------------------------------------------------
-# Chunk hash (for change detection)
-# ---------------------------------------------------------------------------
-
-
-def _chunk_hash(chunks: list[dict[str, Any]]) -> str:
-    """Return a stable SHA-256 hex digest of the concatenated chunk contents."""
-    combined = "\n".join(c["content"] for c in sorted(chunks, key=lambda x: x["id"]))
-    return hashlib.sha256(combined.encode("utf-8")).hexdigest()
-
-
-# ---------------------------------------------------------------------------
-# Compile-state JSON (incremental runs)
-# ---------------------------------------------------------------------------
-
-
-def _load_compile_state(state_path: Path) -> dict[str, str]:
-    """Load ``wiki/.compile-state.json`` → ``{slug: chunk_hash}`` dict."""
-    if not state_path.exists():
-        return {}
-    try:
-        raw = state_path.read_text(encoding="utf-8")
-        data: dict[str, str] = json.loads(raw)
-        return data
-    except (json.JSONDecodeError, OSError):
-        return {}
-
-
-def _save_compile_state(state_path: Path, state: dict[str, str]) -> None:
-    """Persist the compile-state dict to disk."""
-    state_path.write_text(json.dumps(state, indent=2, sort_keys=True), encoding="utf-8")
-
-
-# ---------------------------------------------------------------------------
-# LLM synthesis prompt
-# ---------------------------------------------------------------------------
-
-_SYNTHESIS_PROMPT_TEMPLATE = textwrap.dedent(
-    """\
-    You are a precise academic writer compiling a wiki page on the concept:
-    "{title}"{emphasis_block}
-
-    Use ONLY the source chunks provided below. For every factual claim you make,
-    cite the source chunk inline using the format: [BibKey #locator].
-    Example: "The volume formula is V = L(γ₁)+L(γ₂)+L(γ₃). [Springborn2008 #chunk 16]"
-
-    Do NOT invent facts, formulas, or theorems that are not present in the chunks.
-    If a standard result is not in the chunks, do not include it.
-    Write 3-6 concise paragraphs. Use LaTeX math notation where appropriate ($ … $).
-
-    SOURCE CHUNKS:
-    {chunks_block}
-
-    Now write the wiki page for "{title}":
-    """
-)
-
-
-def _build_synthesis_prompt(
-    concept: Concept,
-    chunks: list[dict[str, Any]],
-) -> str:
-    emphasis_block = ""
-    if concept.emphasis:
-        emphasis_block = f"\n\nEmphasis: {concept.emphasis}"
-
-    chunks_block_lines = []
-    for chunk in chunks:
-        bibkey = chunk.get("bibkey") or chunk["paper_id"]
-        ord_val = chunk.get("ord", "?")
-        chunks_block_lines.append(f"[{bibkey} #chunk {ord_val}]\n{chunk['content'].strip()}\n")
-    chunks_block = "\n---\n".join(chunks_block_lines)
-
-    return _SYNTHESIS_PROMPT_TEMPLATE.format(
-        title=concept.title,
-        emphasis_block=emphasis_block,
-        chunks_block=chunks_block,
-    )
-
-
-# ---------------------------------------------------------------------------
-# Render final page markdown
-# ---------------------------------------------------------------------------
-
-
-def _render_page_markdown(
-    concept: Concept,
-    raw_llm_output: str,
-    claims: list[Claim],
-    compiled_at: datetime,
-) -> str:
-    """Wrap the LLM output in a standard page header and mark ungrounded claims."""
-    ungrounded_texts = {c.text for c in claims if not c.grounded}
-
-    body = raw_llm_output.strip()
-
-    # Mark ungrounded claims inline — replace claim text with ⚠ prefix
-    for text in ungrounded_texts:
-        # Find the claim occurrence and annotate
-        escaped = re.escape(text)
-        body = re.sub(
-            rf"({escaped})",
-            r"⚠ \1",
-            body,
-            count=1,
-        )
-
-    ts = compiled_at.strftime("%Y-%m-%d %H:%M UTC")
-    header = f"# {concept.title}\n\n_Compiled {ts} by `codex wiki compile`_\n\n"
-    return header + body + "\n"
-
-
-# ---------------------------------------------------------------------------
-# Core compile function
-# ---------------------------------------------------------------------------
-
-
-def compile_concept(
-    concept: Concept,
-    chunks: list[dict[str, Any]],
-    *,
-    top_k: int,
-    llm: LLMClient,
-    all_concepts: list[Concept] | None = None,
-    wiki_dir: Path | None = None,
-) -> ConceptPage:
-    """Compile a single concept page.
-
-    Parameters
-    ----------
-    concept:
-        The concept to compile.
-    chunks:
-        Pre-retrieved source chunks for this concept (from :func:`_retrieve_chunks`).
-        Passing chunks explicitly avoids a second retrieve and ensures the stored
-        hash matches the chunks actually used for synthesis.
-
-    1. Synthesise via LLM (Ollama) with per-claim citation format.
-    2. Run Grounding-Guard: mark ungrounded claims as ⚠.
-    3. Inject cross-references to other concepts as [[slug]] links.
-    4. Embed formula chunks if ``formulas`` table is present (graceful).
-
-    Returns a :class:`ConceptPage` with full markdown and claim list.
-    """
-    settings = get_settings()
-    _wiki_dir = wiki_dir or Path(settings.wiki_dir)  # noqa: F841 — kept for future use
-
-    h = _chunk_hash(chunks)
-    prompt = _build_synthesis_prompt(concept, chunks)
-    try:
-        raw_output = llm.generate(prompt, model=settings.wiki_llm_model)
-    except Exception as exc:  # noqa: BLE001
-        logger.warning("LLM unavailable (%s): skipping concept %s", exc, concept.slug)
-        return ConceptPage(concept=concept, markdown="", claims=[], chunk_hash=h)
-
-    claims = _parse_claims(raw_output)
-    claims = _run_grounding_guard(claims, chunks)
-
-    # Graceful: formula-embedding hook (F-09) — currently a no-op (audit C-12).
-    _try_embed_formulas(concept, chunks)
-
-    compiled_at = datetime.now(UTC)
-    # Mark ungrounded claims on the raw body FIRST, then inject cross-refs: a
-    # concept name inside a claim must not stop the ⚠ regex from matching (C-13).
-    markdown = _render_page_markdown(concept, raw_output, claims, compiled_at)
-    markdown = _inject_cross_refs(markdown, all_concepts or [], concept.slug)
-
-    return ConceptPage(
-        concept=concept,
-        markdown=markdown,
-        claims=claims,
-        chunk_hash=h,
-        compiled_at=compiled_at,
-    )
-
-
-def _try_embed_formulas(concept: Concept, chunks: list[dict[str, Any]]) -> None:
-    """Reserved hook for embedding F-09 formula chunks into a page.
-
-    Currently a no-op — formula embedding is not implemented yet. Kept as a stable
-    seam for compile_concept's call site. The previous body issued a per-concept
-    DB round-trip (information_schema lookup) that did nothing (audit C-12).
-    """
-    return
-
-
-# ---------------------------------------------------------------------------
-# compile_all
-# ---------------------------------------------------------------------------
-
-
-def compile_all(
-    *,
-    changed_only: bool = True,
-    top_k: int | None = None,
-    concept_filter: str | None = None,
-    output_dir: str | None = None,
-    llm: LLMClient | None = None,
-) -> CompileReport:
-    """Compile all (or changed) concept pages.
-
-    Parameters
-    ----------
-    changed_only:
-        When ``True`` (default), only recompile concepts whose source-chunk
-        hash differs from the stored state.  ``False`` forces full recompile.
-    top_k:
-        Override ``config.wiki_top_k``.
-    concept_filter:
-        If set, compile only this concept slug.
-    output_dir:
-        Override ``config.wiki_dir``.
-    llm:
-        Injectable LLM client (defaults to :class:`OllamaClient`).
-    """
-    settings = get_settings()
-    wiki_dir = Path(output_dir or settings.wiki_dir)
-    wiki_dir.mkdir(parents=True, exist_ok=True)
-    k = top_k if top_k is not None else settings.wiki_top_k
-
-    concepts_path = wiki_dir / "concepts.yaml"
-    if not concepts_path.exists():
-        # Fall back to sibling concepts.yaml next to wiki/ dir
-        concepts_path = wiki_dir.parent / "wiki" / "concepts.yaml"
-    # Load the FULL concept list — used for cross-reference injection regardless of filter
-    all_concepts = load_concepts(str(concepts_path))
-
-    # Apply filter only to the set of concepts that will be (re-)compiled
-    compile_concepts = (
-        [c for c in all_concepts if c.slug == concept_filter] if concept_filter else all_concepts
-    )
-
-    state_path = wiki_dir / ".compile-state.json"
-    state = _load_compile_state(state_path)
-
-    _llm: LLMClient
-    if llm is not None:
-        _llm = llm
-    else:
-        llm_url = settings.wiki_llm_url or settings.ollama_base_url
-        _llm = OllamaClient(llm_url)
-
-    report = CompileReport()
-
-    for concept in compile_concepts:
-        # Retrieve chunks once — reuse for hash check and synthesis
-        queries = [concept.title] + concept.aliases
-        chunks = _retrieve_chunks(queries, top_k=k)
-        h = _chunk_hash(chunks)
-
-        if changed_only and state.get(concept.slug) == h:
-            report.skipped.append(concept.slug)
-            continue
-
-        page = compile_concept(
-            concept,
-            chunks,
-            top_k=k,
-            llm=_llm,
-            all_concepts=all_concepts,  # always the full list for cross-refs
-            wiki_dir=wiki_dir,
-        )
-
-        # Collect ungrounded claims for the report
-        for claim in page.claims:
-            if not claim.grounded:
-                report.ungrounded.append((concept.slug, claim.text))
-
-        # Detect conflicts between chunks from different bibkeys
-        report.conflicts.extend(_detect_conflicts(concept.slug, chunks))
-
-        # Quarantine check: compute grounding rate.
-        # A page with NO citations at all (total_claims == 0) carries zero grounding
-        # evidence — this also covers the LLM-outage case where compile_concept
-        # returns an empty page — and must NOT be published (audit C-4). Previously
-        # the `total_claims > 0` guard let such pages fall through to wiki/.
-        total_claims = len(page.claims)
-        grounded_claims = sum(1 for c in page.claims if c.grounded)
-        grounding_rate = grounded_claims / total_claims if total_claims > 0 else 0.0
-
-        if total_claims == 0 or grounding_rate < settings.wiki_min_grounding_rate:
-            # Quarantine and do NOT update the compile-state, so a transient failure
-            # (e.g. LLM outage) is retried on the next run instead of being frozen as
-            # "compiled". An empty body is not worth a draft file — just record it.
-            if page.markdown.strip():
-                draft_dir = wiki_dir / "draft"
-                draft_dir.mkdir(parents=True, exist_ok=True)
-                (draft_dir / f"{concept.slug}.md").write_text(page.markdown, encoding="utf-8")
-            report.quarantined.append(concept.slug)
-        else:
-            # Write page to wiki/
-            page_path = wiki_dir / f"{concept.slug}.md"
-            page_path.write_text(page.markdown, encoding="utf-8")
-            state[concept.slug] = page.chunk_hash
-            report.compiled.append(concept.slug)
-
-    _save_compile_state(state_path, state)
-    write_index([_page_summary(slug, wiki_dir) for slug in list(state.keys())])
-    append_log(report, wiki_dir=wiki_dir)
-    return report
-
-
-def _page_summary(slug: str, wiki_dir: Path) -> tuple[str, str]:
-    """Return (slug, title_from_h1) for index generation."""
-    page_path = wiki_dir / f"{slug}.md"
-    title = slug
-    if page_path.exists():
-        first_line = page_path.read_text(encoding="utf-8").splitlines()[0]
-        if first_line.startswith("# "):
-            title = first_line[2:]
-    return (slug, title)
-
-
-# ---------------------------------------------------------------------------
-# write_index
-# ---------------------------------------------------------------------------
-
-
-def write_index(
-    pages: list[tuple[str, str]],
-    *,
-    wiki_dir: Path | None = None,
-) -> None:
-    """Generate ``wiki/index.md`` with [[links]] to all compiled concept pages.
-
-    Parameters
-    ----------
-    pages:
-        List of ``(slug, title)`` tuples.
-    wiki_dir:
-        Path to the wiki directory (defaults to ``config.wiki_dir``).
-    """
-    settings = get_settings()
-    _wiki_dir = wiki_dir or Path(settings.wiki_dir)
-
-    ts = datetime.now(UTC).strftime("%Y-%m-%d %H:%M UTC")
-    lines = [
-        "# Wiki Index",
-        "",
-        f"_Generated {ts} by `codex wiki compile`_",
-        "",
-        "## Concepts",
-        "",
-    ]
-    for slug, title in sorted(pages, key=lambda x: x[0]):
-        lines.append(f"- [[{slug}]] — {title}")
-
-    lines.append("")
-    content = "\n".join(lines)
-    (_wiki_dir / "index.md").write_text(content, encoding="utf-8")
-
-
-# ---------------------------------------------------------------------------
-# append_log
-# ---------------------------------------------------------------------------
-
-
-def append_log(
-    report: CompileReport,
-    *,
-    wiki_dir: Path | None = None,
-) -> None:
-    """Append a run entry to ``wiki/log.md`` (never overwrites).
-
-    Each entry records: timestamp, compiled slugs, skipped slugs,
-    ungrounded claims, and detected conflicts.
-    """
-    settings = get_settings()
-    _wiki_dir = wiki_dir or Path(settings.wiki_dir)
-
-    log_path = _wiki_dir / "log.md"
-    ts = report.ran_at.strftime("%Y-%m-%d %H:%M:%S UTC")
-
-    entry_lines = [
-        f"\n## Run {ts}",
-        "",
-    ]
-
-    if report.compiled:
-        entry_lines.append(f"**Compiled:** {', '.join(report.compiled)}")
-    if report.skipped:
-        entry_lines.append(f"**Skipped (unchanged):** {', '.join(report.skipped)}")
-    if report.ungrounded:
-        entry_lines.append("")
-        entry_lines.append("**⚠ Ungrounded claims:**")
-        for slug, text in report.ungrounded:
-            entry_lines.append(f"- `{slug}`: {text[:120]}")
-    if report.quarantined:
-        entry_lines.append("")
-        entry_lines.append("**QUARANTINED** (grounding rate < threshold → wiki/draft/):")
-        for slug in report.quarantined:
-            entry_lines.append(f"- `{slug}`")
-    if report.conflicts:
-        entry_lines.append("")
-        entry_lines.append("**⚠ Conflicts detected:**")
-        for slug, bk1, bk2 in report.conflicts:
-            entry_lines.append(f"- `{slug}`: conflicting claims in {bk1} vs {bk2}")
-    entry_lines.append("")
-
-    entry = "\n".join(entry_lines)
-
-    # Append-only: open in append mode
-    with log_path.open("a", encoding="utf-8") as fh:
-        if log_path.stat().st_size == 0:
-            fh.write("# Wiki Compile Log\n")
-        fh.write(entry)

conftest.py

@@ -1,17 +0,0 @@
-"""Root conftest — ensures the project root is on sys.path.
-
-Required when the virtual-environment Python is symlinked to an external
-interpreter (e.g. Anaconda) that does not process the venv's .pth files
-automatically.  Adding the project root here makes ``codex`` importable
-regardless of how pytest is invoked (``pytest``, ``python -m pytest``, etc.).
-"""
-
-from __future__ import annotations
-
-import sys
-from pathlib import Path
-
-# Insert the project root so `import codex` always resolves.
-_project_root = str(Path(__file__).parent)
-if _project_root not in sys.path:
-    sys.path.insert(0, _project_root)

docs/SESSION-2026-06-17-open-items.md

@@ -1,59 +0,0 @@
-# Open items — session 2026-06-17 (handoff notice)
-
-Everything still open after this session. Done items are in
-`docs/audit/DATA-QUALITY-2026-06-15.md` (DQ-1..DQ-5 + roadmap R-A,R-B,R-C,R-D,R-F all
-DONE; R-E deferred). This file is only the **loose ends**.
-
-## 1. ✅ RESOLVED — Jetson is online (the tunnel had dropped, not the host)
-- What looked like the Jetson going offline was the **SSH tunnel** breaking, not the
-  host. The Jetson (`jetson.local`) is online. Re-open the tunnel with keepalive:
-  `ssh -o ServerAliveInterval=15 -f -N -L 5433:localhost:5432 alfred@jetson.local`.
-  Better still: run long ops in tmux-on-Jetson (item #4) to avoid the tunnel entirely.
-
-## 2. ✅ DONE — R-F accent re-clean
-- R-F is functionally DONE (section signal 34/329 → **314/329 = 95 %**). The remaining
-  cosmetic accent/tie artifacts in live `chunks.section` labels were re-cleaned in place
-  once the tunnel was back: applied `codex.quality._clean_title` to every label
-  containing `\` or `~` — **8 labels fixed**, no re-embed. Verified **0 artifacts**
-  remain. R-F fully complete.
-
-## 3. ✅ READY TO MERGE — all branches integrated + validated
-All five feature branches are merged (**zero conflicts**) into **`integration/roadmap`**
-(off `audit/loop-1`) and validated green: **391 tests pass, ruff + ruff-format + mypy
-clean, tree clean**. This is the single merge candidate for `main`.
-
-Contents (DQ-1..DQ-5 + R-A..R-F + infra docs):
-| Source branch | Contents |
-|---|---|
-| `audit/loop-1` (base) | DQ-1..DQ-5, **R-A** (GROBID backfill + native arm64 GROBID), this notice |
-| `feat/section-labels` (incl. `feat/tex-ingest`) | **R-C** (.tex ingest) + DQ-5 arXiv-id fix + gzip fallback + **R-F** (section labels) |
-| `feat/crossref-references` | **R-B** (Crossref refs) + GROBID-on-demand tweak |
-| `feat/graph-coverage-warning` | **R-D** (citing-coverage warning) |
-| `chore/infra-ssh-stability` | infra TODO (#4) |
-
-**To land on `main`** (user-triggered):
-```
-git checkout main && git merge --no-ff integration/roadmap
-```
-…or open a PR `integration/roadmap` → `main`. The individual feature branches are
-preserved unchanged for per-feature review if preferred.
-
-**Cleanup — branches prunable AFTER the merge lands** (not deleted here; destructive):
-- Already in `main`: `fix/cli-readme-drift`, `fix/d04-d05-ingest-fixes`.
-- Folded into `integration/roadmap`: the 5 feature branches + `audit/loop-1`.
-- Pre-session / stale (verify before pruning): `feat/F-15-literature-graph`,
-  `feat/F-16-chunk-quality`, `fix/audit-wave-1`/`-2`/`-3`, `scratch/spike-embed`.
-- Working tree is clean — only gitignored `.venv`/caches/`.env`; no stray tracked files.
-
-## 4. Infra TODO — tmux-on-Jetson for stable long ops (documented, not implemented)
-- `docs/infra/jetson-ssh-stability.md` (branch `chore/infra-ssh-stability`). The
-  Mac↔Jetson tunnel dropped repeatedly on multi-minute writes all session. Fix: run long
-  ops **on the Jetson in `tmux`** (DB is localhost there → no tunnel; survives SSH drops);
-  `autossh` for interactive tunnels. Open.
-
-## 5. Known limitations (documented, NOT action items)
-- `math/0306167` (old AMS-TeX, no `\section{}`) stays `section=body` — inherent.
-- The 2 edited-volume books yielded 0 GROBID-parsed DOI/arXiv refs in the R-A sweep —
-  inherent (per-chapter refs, no clean end-bibliography).
-- **DQ-6** (upstream PDF→`.txt` extraction fidelity, done outside codex-py) — flagged
-  earlier, largely **mooted** by R-A/R-C ingesting from PDF/`.tex` directly.

docs/adr/ADR-F12-wiki-compile.md

@@ -1,127 +0,0 @@
-# ADR-F12 — Wiki-Compile: Grounded Concept Pages
-
-**Status:** IMPL (GO after Spike-Session 2026-06-08; Gate-Fixes 2026-06-14)
-**Owners:** F-12 Worker (Sonnet)
-**Last updated:** 2026-06-14
-
----
-
-## Context
-
-F-12 adds a wiki-compile layer on top of the RAG substrate: for each concept
-in `wiki/concepts.yaml`, retrieve Top-K chunks via hybrid dense+FTS search,
-synthesise a concept page via local LLM (Ollama / qwen2.5:7b), and write
-the result to `wiki/<slug>.md`.
-
-Primary risk: **hallucination**. An LLM can invent theorems, formulas, and
-citations that are not in the source corpus. If left unchecked, the wiki
-becomes an authoritative-looking source of misinformation.
-
-Secondary risk: **undetected quarantine escape**. A page with many ungrounded
-claims must not appear in the main `wiki/` directory alongside verified pages.
-
----
-
-## Spike Result (2026-06-08)
-
-Manual evaluation on 3 concepts (discrete-conformal-map, circle-packing,
-lobachevsky-function), corpus = 3 papers (Springborn2008, BPS2015, Luo2004),
-95 chunks:
-
-| Metric | Result |
-|---|---|
-| Grounding rate | ~0.95 (38/40 sampled claims grounded) |
-| Hallucination rate | ≤ 5 % (no invented theorem, no false formula in sample) |
-| False ungrounded | ~5 % (legitimate claims not matched due to paraphrase) |
-
-Conclusion: **GO** — grounded synthesis is viable on this corpus.
-
----
-
-## Decision
-
-### Grounding Guard (implemented in `codex/wiki.py: _run_grounding_guard`)
-
-**Approach:** Content-5-gram match in content-word space.
-
-1. **Sentence granularity:** Extract the *last sentence* of each claim text
-   (split on `.!?`, take the last non-empty fragment). Citations annotate the
-   immediately preceding sentence; checking the whole paragraph would allow
-   a hallucinated block to be grounded by a single phrase at its end.
-
-2. **Content-5-gram:** Remove stopwords from both claim sentence and chunk
-   text. Require 5 consecutive non-stopword tokens from the claim sentence
-   to appear as 5 consecutive non-stopword tokens in the chunk's content-word
-   stream (same bibkey). This prevents trivial bypass via common scientific
-   phrases ("the discrete conformal map" = only 2 content words → ungrounded).
-
-3. **Short-sentence fallback:** Claims with < 5 content words in the last
-   sentence are marked **ungrounded** by default (insufficient signal).
-
-**Adversarial test (Review-Gate Finding):**
-> "The Riemann hypothesis is proven by combining Hodge theory with quantum
-> entanglement. Furthermore P=NP holds for hyperbolic tetrahedra.
-> The discrete conformal map [Springborn2008 #chunk 0]"
-
-→ Last sentence: "The discrete conformal map" → 3 content words < 5 →
-`grounded = False`. Bypass closed.
-
-### Quarantine Mechanism (implemented in `compile_all`)
-
-Pages with `grounding_rate < wiki_min_grounding_rate` (default 0.5) are:
-- Written to `wiki/draft/<slug>.md` instead of `wiki/<slug>.md`
-- Tracked in `CompileReport.quarantined`
-- Logged as `QUARANTINED` in `wiki/log.md`
-
-`codex wiki check` exits with:
-- `0` — all claims grounded, `wiki/draft/` empty
-- `1` — at least one `⚠` in `wiki/*.md`
-- `2` — `wiki/draft/` non-empty (quarantined pages present)
-
-Exit 2 takes priority over Exit 1.
-
-Config: `WIKI_MIN_GROUNDING_RATE` (env/`.env`), default `0.5`.
-
-### LLM Graceful Degradation
-
-`compile_concept` wraps `llm.generate()` in a `try/except`. On any exception
-(httpx.ConnectError, timeout, …), it logs a warning and returns an empty
-`ConceptPage` (no crash). An empty page has 0 claims → grounding rate 0.0 →
-quarantine path.
-
-### Conflict Detection (MVP)
-
-`_detect_conflicts` uses an adversative keyword heuristic ("but", "however",
-"in contrast", "contradicts", …) across pairs of chunks from different bibkeys.
-Results in `CompileReport.conflicts`. This is a signal, not a blocker —
-conflicts are logged but do not block page compilation.
-
-### Cross-Reference Injection
-
-- Inline-code spans (`` `…` ``) are protected by null-byte placeholders
-  before `_inject_cross_refs` runs, then restored. This prevents concept
-  titles inside code from being rewritten.
-- `all_concepts` passed to `compile_concept` is always the **full** concept
-  list from `concepts.yaml`, regardless of `--concept` filter. The filter
-  controls which pages are regenerated, not which cross-refs are resolved.
-
----
-
-## Fallback
-
-If grounding rate drops below 50% (config threshold), pages land in
-`wiki/draft/` (extraktiver Fallback). Human review required before promotion
-to `wiki/`.
-
-For `wiki_min_grounding_rate = 0.0`, quarantine is disabled (all pages written
-to `wiki/` regardless of grounding). Not recommended for production.
-
----
-
-## Consequences
-
-- R-27 met: Grounding ≥ 90% on spike corpus; Halluzinations-Rate ≤ 5%.
-- R-24 partially met: conflict detection is keyword-heuristic (MVP), not
-  semantic. Full semantic conflict detection deferred to F-13.
-- No DB schema changes (F-12 constraint). All state in `wiki/` directory.
-- Ollama endpoint required at compile time; graceful degradation if unavailable.

docs/adr/ADR-F13-synthesis-engine.md

@@ -1,141 +0,0 @@
-# ADR-F13 — Synthesis / Lead-Engine
-
-**Status:** IMPL (GO after informal spike; Gate-Fixes 2026-06-14, 2 Opus passes)
-**Owners:** F-13 Worker (Sonnet)
-**Last updated:** 2026-06-15
-
----
-
-## Context
-
-F-13 adds a lead-generation layer on top of the F-12 wiki/RAG substrate.
-Where F-12 produces *descriptive* concept pages (grounded, no new claims),
-F-13 produces *research leads*: structured hypotheses about connections, gaps,
-improvements, and conjectures over the corpus.
-
-Primary risk: **fact-leak** — a model-generated conjecture being presented as
-an established result. If a conjecture appears in `wiki/` without the
-`status: unverified` marker, it becomes indistinguishable from a reviewed claim.
-
-Secondary risk: **grounded leads that are not actually grounded** — an LLM
-can cite a bibkey without the claim appearing in the cited chunk. This is
-the same hallucination vector as F-12, but now in the context of novel claims.
-
----
-
-## Spike Result (2026-06-14)
-
-Informal spike run by Opus on the initial ingest corpus (≥ 3 papers / ≥ 95
-chunks available at spike time; full 29-paper ingest completed in parallel).
-
-| Criterion | Target | Result |
-|-----------|--------|--------|
-| Grounded-lead precision | ≥ 60 % | GO — usable connection + gap leads produced |
-| ≥ 1 useful conjecture | ≥ 1 non-trivial | GO — ≥ 1 brauchbare Konjektur |
-| Zero-fact-leak (hard) | 0 conjectures presented as facts | PASS — hard-enforced in code |
-
-**GO** — synthesis is viable. The conjecture generator produces novel
-directions at a useful rate; the hard invariants (unverified status, quarantine
-directory) eliminate the fact-leak risk at the code level, not just as policy.
-
-Opus reviewer approved implementation in 2 passes (gate-fix pass addressed
-non-empty provenance enforcement and gap-lead edge case). 215 tests green,
-ruff + mypy clean.
-
----
-
-## Decision
-
-### Four-Stage Lead Taxonomy
-
-| Stage | Kind | Grounded? | Output |
-|-------|------|-----------|--------|
-| 2 | `connection` | yes — content-5-gram check | `leads/grounded/` |
-| 3 | `gap` | yes — grounded on absence | `leads/grounded/` |
-| 4 | `improvement` | yes — against cited chunks | `leads/grounded/` |
-| 5 | `conjecture` | no — by definition | `leads/conjectures/` |
-
-### Grounding Discipline (reuse of F-12 guard)
-
-Stages 2–4 reuse `_run_grounding_guard` and `_parse_claims` from `codex/wiki.py`
-directly. The same content-5-gram check applies: a claim must have ≥ 5 consecutive
-non-stopword tokens from the last sentence appear in the cited chunk. Ungrounded
-claims get a `⚠` prefix; a lead whose grounded-ratio falls below
-`synthesis_min_grounded_ratio` (config, default used by tests) is dropped
-entirely — not written to disk.
-
-### Zero-Fact-Leak Enforcement (Hard Invariant)
-
-Conjectures are enforced as unverified at three independent levels:
-
-1. **`status` field** — `propose_conjectures` hard-sets `status="unverified"`
-   and `confidence=0.0`. No caller can override this without modifying the
-   generator directly.
-
-2. **Falsification gate** — `_parse_conjecture_output` returns `None` if the
-   LLM response is missing the `Validation:` section. Conjectures without a
-   falsification path are silently dropped. This makes the invariant
-   self-enforcing: an LLM that omits the validation block produces no conjecture.
-
-3. **Directory routing** — `write_leads` routes `kind="conjecture"` exclusively
-   to `leads/conjectures/`. A runtime check raises `RuntimeError` if a conjecture
-   would be written to `leads/grounded/` or anywhere under `wiki/`.
-   `test_quarantine.py` asserts this routing for every output call.
-
-### Gap Leads: Grounded on Absence
-
-Stage-3 gaps have two signals:
-
-* **Coverage-map gap** — a topic retrieved by < `synthesis_gap_min_coverage`
-  bibkeys. The LLM is asked to articulate what is *present* (citable) and what
-  is absent. The absence itself needs no citation; the surrounding context does.
-* **Code-vs-corpus gap** — a `@cite` bibkey in the C++ library that has no
-  matching paper in the corpus. These are created without LLM (the absence is
-  structural, not semantic) and carry `confidence=1.0`.
-
-### Conjecture Provenance (Mandatory Seed Citation)
-
-`propose_conjectures` always populates `provenance` from the seed chunks used
-to generate the conjecture. A conjecture with an empty `provenance` list is
-rejected (the `Lead.__post_init__` ValueError guard). This gives reviewers a
-traceable path from the novel claim back to the literature that seeded it —
-without implying those chunks support the claim.
-
-### LLM Graceful Degradation
-
-`_safe_llm_generate` wraps every LLM call in a `try/except` and returns `""`
-on any error. This matches the F-12 degradation rule: an unreachable Ollama
-endpoint produces an empty lead list, not a crash.
-
-### Persistence: Files, No Schema Migration
-
-Leads are written as Markdown files with a fenced JSON front-matter block.
-No `schema.sql` migration is needed. `leads/INDEX.md` is a rebuild-on-write
-summary (not append-only on disk, but the effect is append-only from the
-user's perspective because older files are preserved on re-runs).
-
----
-
-## Fallback
-
-If zero grounded leads are produced (e.g. LLM unreachable, corpus too small):
-`write_leads([])` writes only an empty `INDEX.md`. The synthesis layer does
-not block downstream features (F-14 MCP server exposes `synthesis_browse`
-which gracefully handles an empty `leads/` directory).
-
-For conjecture quality below the minimum useful threshold, the `--conjectures`
-flag can be omitted: the `codex synthesis leads` command runs only stages 2–4.
-
----
-
-## Consequences
-
-- R-28: Lead/Provenance datamodel — `codex/synthesis.py` (43 synthesis tests green)
-- R-29: `find_gaps` — coverage-map + code-vs-corpus signal
-- R-30: `find_connections` + `find_improvements` — grounded, ⚠ on failure
-- R-31: `propose_conjectures` — `status=unverified`, mandatory provenance + validation
-- R-32: Quarantine invariant — enforced at 3 levels, tested in `test_quarantine.py`
-- R-33: `codex synthesis leads/conjectures/report` CLI — `test_cli.py` green
-- R-34: **this document** — GO confirmed, zero-fact-leak enforcement documented
-- No DB schema changes. All lead state in `leads/` directory (gitignored by default).
-- Ollama endpoint required for stages 2–5; graceful empty output if unavailable.

docs/adr/ADR-F15-literature-graph.md

@@ -1,204 +0,0 @@
-# ADR-F15 — Literature Graph / Citation PageRank
-
-**Status:** IMPL — GO after live-corpus spike 2026-06-15. See **Audit Update
-(2026-06-15)** at the end: the ID-format mismatch is now resolved in code
-(audit C-1, C-7) and the Spike Result numbers below predate that resolution.
-**Owners:** F-15 Worker (Sonnet)
-**Last updated:** 2026-06-15 (audit addendum)
-
----
-
-## Context
-
-F-15 adds a citation graph layer on top of the corpus DB.  The goals are:
-
-1. **Hub detection** — identify which papers are most central in the citation
-   network (PageRank) so that `codex search paper --cite-boost` can
-   up-weight them in dense retrieval.
-2. **Bibliographic coupling / co-citation** — surface related papers that
-   share many references or are frequently cited together.
-3. **Dangling-citation discovery** — enumerate works cited by the corpus but
-   not yet ingested, serving as an acquisition queue.
-
-Primary risk: **degenerate PageRank** on a small, sparse corpus producing
-scores so flat that the boost has no practical effect or — worse — distorts
-results by promoting low-quality dangling works.
-
----
-
-## Spike Result (2026-06-15)
-
-Run on the live Jetson corpus: **29 ingested papers, 590 citation edges,
-495 total graph nodes, 478 dangling nodes**.
-
-### Hub-Paper Ranking
-
-| Rank | OpenAlex ID | Year | Authors | Title | PR |
-|------|-------------|------|---------|-------|----|
-| 1 | W1974956622 | 1993 | Pinkall, Polthier | Computing Discrete Minimal Surfaces and Their Conjugates | 0.002282 |
-| 2 | W1964119857 | 1962 | Ford, Fulkerson | Flows in Networks | 0.002228 |
-| 3 | W312477465  | 1968 | Trudinger | Remarks on conformal deformation of Riemannian structures | 0.002204 |
-| 4 | W1595775335 | 1960 | Yamabe | On a deformation of Riemannian structures on compact manifolds | 0.002204 |
-| 5 | W2023551584 | 1996 | Cooper, Rivin | Combinatorial scalar curvature and rigidity of ball packings | 0.002204 |
-| 6 | W2159531493 | 1984 | Schoen | Conformal deformation of a Riemannian metric to constant scalar curvature | 0.002204 |
-| 7 | **W2163787581** | **2007** | **Bobenko, Springborn** | **A Discrete Laplace–Beltrami Operator for Simplicial Surfaces** | **0.002195** |
-| 8 | W3099917509 | 2012 | Mercat | Discrete Riemann Surfaces and the Ising Model | 0.002190 |
-| 9 | W2021879624 | 2008 | Mullen, Tong, et al. | Spectral Conformal Parameterization | 0.002164 |
-| 10 | W1976584051 | 2002 | Desbrun, Meyer, et al. | Intrinsic Parameterizations of Surface Meshes | 0.002159 |
-
-### Validation Against Manual Expectation
-
-| Criterion | Target | Result |
-|-----------|--------|--------|
-| Bobenko or Springborn in top-10 hubs | ✓ expected hub | **PASS — Bobenko & Springborn 2007 at rank 7** |
-| Pinkall & Polthier 1993 prominent | ✓ foundational DGP paper | **PASS — rank 1** |
-| Score flatness acceptable | < 2× spread over top-10 | PASS — 5.7 % spread (0.002282 → 0.002159) |
-| Graceful small-corpus warning | shown when < 15 in-KB papers | **PASS — shown at 29 in-KB papers** (< `graph_min_corpus_size=15` not triggered; Springborn 2008 is not in-KB so not visible as "ingested hub") |
-
-**GO** — PageRank produces a plausible field-specific ranking.  The top hits
-(Pinkall/Polthier, Yamabe, Schoen, Bobenko/Springborn) are exactly the
-foundational works a DGP researcher would expect to see elevated.
-
----
-
-## Decision
-
-### Graph Construction
-
-`build_citation_graph` builds an in-memory `nx.DiGraph` from the `citations`
-table: `citing_id → cited_id`.  No schema migration needed.
-
-- `citing_id` stores DOIs (matches `papers.id`).
-- `cited_id` stores OpenAlex IDs (external references).
-
-**Consequence:** in the current corpus, ingested papers do not appear as
-in-degree targets of each other — inter-KB citations are structurally
-invisible because the `cited_id` column uses OpenAlex IDs while `papers.id`
-uses DOIs.  The graph is therefore bipartite-like (29 DOI-nodes →
-478 OpenAlex-nodes), and all PageRank mass concentrates on the dangling
-OpenAlex-nodes cited by many of our 29 papers.
-
-This is acceptable for the `--cite-boost` use-case: dangling hub papers are
-*foundational works* our corpus cites repeatedly, so boosting search results
-that match them is desirable.
-
-### PageRank Graceful Degradation
-
-`citation_pagerank` returns a **uniform distribution** when the graph has
-fewer than 5 nodes (`_MIN_PAGERANK_NODES`) and logs a warning.
-
-The CLI `graph report` emits a separate user-visible warning when the count
-of in-KB papers is below `graph_min_corpus_size` (default 15) — because at
-that point the *hub ordering* (which uses PageRank over all 495 nodes) is
-still valid, but its *discriminative power* is limited by the few citing
-sources.
-
-At 29 ingested papers, `graph_min_corpus_size=15` is satisfied; the score
-spread is narrow (5.7 %) but the ranking order is meaningful.
-
-### cite-boost Formula
-
-```
-boosted_distance = distance / (1.0 + alpha * pr_score)
-```
-
-`alpha = 0.3` (config: `graph_cite_boost_alpha`).  Dividing reduces cosine
-distance for high-PR papers (lower = closer in pgvector).  Multiplying
-would invert the boost — this was the critical bug corrected during the
-F-15 review gate.
-
-At `alpha=0.3` and the observed PR scores (~0.002), the maximum boost is
-approximately 0.06 % per unit score — effectively a tie-breaker on equally
-relevant papers rather than a hard re-ranking.  This is intentional: we do
-not want citation authority to override semantic relevance for queries that
-genuinely target niche papers.
-
-### Dangling Citations as Acquisition Queue
-
-`dangling_citations(graph, known_ids)` returns the 478 OpenAlex IDs that
-are cited but not ingested.  `codex graph report` surfaces these so the
-researcher can prioritize which papers to add to the corpus next.
-
-The top hubs (rank 1–10) are the highest-value acquisition targets: adding
-Pinkall & Polthier 1993, Yamabe 1960, and Schoen 1984 would improve both
-retrieval quality and PageRank discrimination.
-
----
-
-## Fallback
-
-If the corpus has < `_MIN_PAGERANK_NODES` (5) nodes: `citation_pagerank`
-returns uniform scores; `--cite-boost` has no effect (each paper is boosted
-equally); no crash.
-
-If `--cite-boost` is omitted: search works as F-03/F-04 baseline.
-
-If the `citations` table is empty: graph has 0 nodes; `graph report` prints
-"Graph is empty" and exits 0; all other commands degrade gracefully.
-
----
-
-## Known Limitation: ID-Format Mismatch
-
-> **RESOLVED (audit 2026-06-15) — see Audit Update below.** This no longer holds:
-> `build_citation_graph` resolves OpenAlex `cited_id`s to the canonical
-> `papers.id` via the shared `RESOLVED_CITATIONS_SQL` (C-1) and ingest stores
-> canonical ids (C-7), so inter-KB citations *are* now edges. Original text kept
-> for the record.
-
-`cited_id` uses OpenAlex IDs; `papers.id` uses DOIs.  Cross-citations
-between ingested papers are therefore not represented as graph edges.
-This is a D-05-class issue (same root cause as the `citing_id`/`openalex_id`
-FK mismatch fixed in PR #9).  A future migration could add an `openalex_id`
-column to `papers` and populate `cited_id` cross-links; but the benefit is
-small while the corpus is < 100 papers.
-
----
-
-## Consequences
-
-- R-41: `build_citation_graph` — in-memory DiGraph, no schema change ✓
-- R-42: `citation_pagerank` + `codex search paper --cite-boost` ✓
-- R-43: `find_related` (bibliographic coupling) + `find_co_cited` ✓
-- R-44: `codex graph report [--top-n N] [--json]` ✓
-- R-45: graceful degradation < 5 nodes: uniform scores, warning ✓
-- R-46: **this document** — GO; Bobenko+Springborn 2007 at rank 7 validates hub detection; score flatness expected at corpus size 29; increase to > 100 papers for stronger PageRank differentiation
-
----
-
-## Audit Update (2026-06-15)
-
-A post-hoc audit of the shipped F-15 code found that the live-corpus spike above
-was measured on a graph the current code no longer produces:
-
-- **ID-format mismatch resolved.** `build_citation_graph` resolves OpenAlex
-  `cited_id`s to the canonical `papers.id` through the shared
-  `RESOLVED_CITATIONS_SQL`, now also used by `codex.discover` so the two cannot
-  diverge (audit C-1). Ingest stores the caller's canonical id as `papers.id`
-  instead of OpenAlex's URL-form doi (audit C-7). Inter-KB citations are
-  therefore represented as edges — the "Known Limitation" above is obsolete.
-- **Spike re-measured after migration (2026-06-15).** The corpus was re-ingested
-  onto canonical bare ids (29 papers, all bare — verified `url_form_ids = 0`), and
-  the graph re-run on the post-migration corpus:
-
-  | Metric | Original spike (pre-fix) | Post-migration (re-measured) |
-  |--------|--------------------------|------------------------------|
-  | Ingested papers | 29 | 29 |
-  | Graph nodes | 495 | 489 |
-  | Graph edges | 590 | 590 |
-  | Dangling nodes | 478 | 472 |
-  | Top-10 PR spread | 5.7 % | 5.4 % |
-  | Rank-1 hub | W1974956622 (dangling) | **Pinkall/Polthier 1993 — in-KB (resolved)** |
-  | Bobenko/Springborn 2007 | rank 7 | rank 7 |
-
-  The resolution fix's signature is visible: Pinkall/Polthier 1993 now resolves to
-  its canonical DOI and rises to **rank 1 as an in-KB hub** (previously a dangling
-  OpenAlex node). Bobenko/Springborn 2007 holds rank 7, and the foundational works
-  (Pinkall/Polthier, Yamabe, Schoen, Bobenko/Springborn) still surface. **GO
-  re-affirmed** on the shipping topology.
-
-  Migration footnotes: two papers initially survived in URL form because the
-  id-keyed upsert did not catch a `papers.openalex_id` collision, and `2305.10988`
-  failed on a `papers.bibkey` collision with `2310.17529` (both →
-  `BobenkoLutz2023`). Same upsert gap — resolved by the bibkey-retry (audit C-15)
-  plus a delete + re-ingest of the two stragglers; corpus is now 29/29 canonical.

docs/audit/AUDIT-2026-06-15-full-repo.md

@@ -1,616 +0,0 @@
-# AUDIT — Full Repository (iterative)
-
-**Auditor:** Audit-Loop (Opus)
-**Started:** 2026-06-15
-**Scope:** entire `main` tree, audited iteratively module-by-module.
-**Companion doc:** [`AUDIT-2026-06-15-loop-1.md`](AUDIT-2026-06-15-loop-1.md) —
-the F-15 / dev-loop-1 findings (S-1…U-4). This document continues the **same
-finding-ID register** (so `C-4` here follows `C-3` there) and covers the rest of
-the repo.
-
-> Severity: **HIGH** fix before relying · **MED** fix soon · **LOW** polish ·
-> **INFO** accepted/no-action. Findings only — remediation planned separately.
-
-## Coverage tracker
-
-| Part | Area | Status |
-|------|------|--------|
-| (loop-1) | F-15 graph, db/schema, ingest, discover, mcp, provenance-scan, cli(search+graph), config(graph) | ✅ documented |
-| **A** | Grounding guard (`wiki.py` `_parse_claims`/`_run_grounding_guard`, quarantine) — shared by `synthesis.py` | ✅ documented |
-| **B** | `sources/` (arxiv, openalex, semanticscholar) | ✅ documented |
-| **C** | `parsing/` (grobid, nougat, mathpix, figures, tex) | ✅ documented |
-| **D** | `synthesis.py` orchestration (find_connections/gaps/improvements, conjectures, write_leads) | ✅ documented |
-| **E** | `wiki.py` rest (retrieve, compile_concept, cross-refs, index/log, embed) | ✅ documented |
-| **F** | `embed.py`, `quality.py`, `models.py` | ✅ documented |
-| **G** | `cli.py` rest (ingest/wiki/synthesis/provenance commands), `config.py` rest | ✅ documented |
-
----
-
-## Part A — Grounding guard (the "zero-fact-leak" core)
-
-The guard lives in `wiki.py` (`_parse_claims:216`, `_run_grounding_guard:299`)
-and is imported by `synthesis.py:57-59`, so **one implementation gates both**
-the wiki pages and the synthesis leads. A claim is "grounded" iff the **last
-sentence** before its `[BibKey #loc]` citation contains a run of 5 consecutive
-non-stopword tokens that appears as a substring of any chunk of the same bibkey.
-
-### C-4 — Zero-claim pages bypass quarantine and publish as "compiled" · **HIGH**
-- **Evidence:** `wiki.py:730` — `if total_claims > 0 and grounding_rate < threshold:`
-  quarantines; the `else` (`:737-742`) **publishes to `wiki/` and records the
-  hash** as successfully compiled.
-- **Bug:** when the LLM output has **no parseable citations** (`total_claims == 0`,
-  so `grounding_rate == 0.0`), the `total_claims > 0` guard is false → it takes
-  the *publish* branch. The quarantine gate is skipped exactly when grounding
-  evidence is **weakest** (zero citations).
-- **Trigger paths:** (a) the LLM emits uncited prose; (b) **LLM outage** — the
-  module header guarantees "LLM unavailable → empty ConceptPage, no crash"
-  (`wiki.py:13-14`), so a transient Ollama outage writes near-empty stub pages to
-  `wiki/`, marks them compiled, and stores their hash. On the next
-  `changed_only` run the hash matches → the stub is **skipped forever** until a
-  forced recompile. A research wiki silently fills with empty/uncited pages.
-
-### C-5 — "Every claim is grounded" holds only for the last cited sentence · **MED**
-- **Evidence:** `_CLAIM_RE` (`wiki.py:211-213`) captures `text = [^\[]+?` (the
-  span back to the previous `]`/start); `_last_sentence` (`:279-291`) then keeps
-  only the final sentence; `_run_grounding_guard` checks that sentence only.
-- **Two holes in the guarantee:**
-  1. **Paragraph passengers** — any sentence that is *not* the last one before a
-     citation is never grounding-checked and never `⚠`-marked. "Hallucination one.
-     Hallucination two. Grounded tail [Bib]." passes with only the tail verified.
-  2. **Uncited text is invisible** — a sentence with no `[BibKey]` is not a Claim
-     at all (`_parse_claims` only yields cited spans), so it is never grounded
-     nor marked. The LLM can emit fully uncited assertions into a published page.
-- **Impact:** the effective invariant is "*the last sentence immediately before
-  each citation has lexical overlap with the cited paper*" — materially weaker
-  than the N-02 "zero-fact-leak enforcement" framing. The module **header is
-  honest** ("Substring-Match MVP", `wiki.py:4,13`); the commit/ADR language is
-  not. Recommend aligning the claim or strengthening the guard (check every
-  sentence; flag uncited sentences).
-
-### C-6 — Substring match does not enforce the documented "5 consecutive tokens" · **LOW**
-- **Evidence:** `wiki.py:347-349` — `gram = " ".join(content[i:i+5])`;
-  `if any(gram in src for src in sources)` where `src` is the space-joined
-  content-word string of the chunk.
-- **Issue:** a *substring* test on the joined string lets the first/last gram
-  token match as a **prefix/suffix of a longer chunk word** (e.g. gram
-  `"cat dog …"` substring-matches inside `"scat dogma …"`). The docstring claims
-  matching "as 5 consecutive non-stopword **tokens**" — true sequence matching
-  would compare token lists with a sliding window (or add boundary sentinels).
-  Low real-world impact at 5-gram length, but the claimed invariant is not what
-  the code enforces.
-
-### R-1 — Tokenizer brittleness causes false-negative over-quarantine · **MED**
-- **Evidence:** `_content_words` (`wiki.py:294-296`) lowercases and `.split()`s on
-  whitespace only — **no punctuation stripping**. `_last_sentence` splits on
-  `.!?` (`:276`).
-- **Impact:** punctuation stays glued to tokens, so a faithful paraphrase grounds
-  only on *exact* surface form — `"volume,"` ≠ `"volume"`, `"L(gamma1)"` must
-  match character-for-character, and a trailing decimal like `"V = 1.5"` makes
-  `_last_sentence` return the fragment `"5"` (split at the `.`), which then has
-  `< 5` content words → **forced ungrounded** (`:339-344`). Math-heavy and terse
-  claims are systematically marked ungrounded even when verbatim, inflating the
-  ungrounded ratio and pushing real pages into quarantine (and into C-4's
-  blind spot when it drives `total_claims` interactions). The passing test
-  `test_grounding_guard_…` only succeeds because its fixture reproduces the chunk
-  text *exactly*.
-
-### R-2 — Reference-list filter can drop theorem-dense chunks · **LOW**
-- **Evidence:** `_retrieve_chunks` (`wiki.py:188-199`) skips a chunk when
-  `>60 %` of its lines match `^\s*(\[\d+\]|[A-Z][a-z]+,?\s+[A-Z]\.)`.
-- **Issue:** the second alternative (`[A-Z][a-z]+,?\s+[A-Z]\.`) also matches prose
-  like `"Theorem A."`, `"Lemma B."`, `"Section C."`. A chunk formatted as a list
-  of theorems/lemmas can exceed 60 % and be discarded as a "reference list" —
-  silently dropping exactly the mathematical content the wiki wants.
-
-### R-3 — Conflict detection has near-zero precision · **LOW**
-- **Evidence:** `_detect_conflicts` (`wiki.py:367-…`) flags a pair `(bk1, bk2)` if
-  **each** has *any* chunk containing a hedging keyword
-  (`but|however|in contrast|…`, `_CONFLICT_KEYWORDS:361-364`).
-- **Issue:** it never checks the keywords relate to a shared topic or to each
-  other. In a corpus where most papers contain "however"/"but" somewhere, this
-  flags ~O(n²) pairs as "conflicting" — noise surfaced in `CompileReport`.
-  Labeled "heuristic signal only", but as-is it is not actionable.
-
-**Part A net:** 1 HIGH (C-4), 2 MED (C-5, R-1), 3 LOW (C-6, R-2, R-3). The guard
-is a reasonable lexical MVP, but (a) it has a concrete publish-bypass bug for
-zero-citation output, and (b) the "zero-fact-leak" framing overstates what a
-last-sentence substring check can guarantee.
-
----
-
-## Part B — `sources/` (external API clients)
-
-Solid foundations: all three clients set httpx timeouts; OpenAlex and S2 retry
-429/5xx with exponential back-off (tenacity); S2 enforces a ≥1 req/s floor with a
-monotonic clock under a lock. The findings are about **ID format consistency**
-and a couple of dead/inert paths.
-
-### C-7 — `papers.id` is stored in OpenAlex *URL* form, breaking the bare-ID contract · **HIGH**
-- **Evidence:** `openalex.py:94` — `_map_paper` sets
-  `id = data.get("doi") or data.get("id") or ""`. Real OpenAlex returns `doi` as
-  a **full URL** (`https://doi.org/10.x`) and `id` as `https://openalex.org/W…`
-  (the repo's own test confirms the URL form for `openalex_id`,
-  `test_openalex.py:92`). `ingest.py:82` takes `paper.id` from this unchanged and
-  upserts it as `papers.id` (`ingest.py:138`).
-- **Why it's currently masked:** the test fixture `_SAMPLE_WORK["doi"]` is a
-  **bare** DOI `"10.1145/3592430"` (`test_openalex.py:18`) — not the URL form real
-  OpenAlex emits — and **no test asserts `paper.id`** at all. So the suite is
-  green while production stores `papers.id = "https://doi.org/10.1145/3592430"`.
-- **Impact:**
-  1. **Bare-ID lookups fail.** `ingest_all.sh` ingests `"10.1145/1964921.1964997"`,
-     but `papers.id` becomes the URL; `codex graph related 10.1145/…`,
-     `discover citing 10.1145/…`, search-by-id, etc. find nothing.
-  2. **Second ID-mismatch axis (compounds C-1).** `cited_id` from the S2/GROBID
-     paths is a **bare** DOI/arXiv (`semanticscholar.py:98-99`,
-     `ingest.py:187`), so a bare-DOI reference can never equal a URL-form
-     `papers.id` — bare-DOI cross-citations stay "dangling" regardless of the
-     F-15 W-ID resolution (which only matches `papers.openalex_id`, the URL form).
-- **Action:** confirm with `SELECT id FROM papers LIMIT 5` on the Jetson DB; if
-  URL-form, normalize `papers.id` to a canonical bare form at ingest (and the
-  `cited_id` writers to match), then add a representative fixture + `paper.id`
-  assertion (see T-3).
-
-### T-3 — OpenAlex test fixture is unrepresentative and skips `paper.id` · **MED**
-- **Evidence:** `_SAMPLE_WORK["doi"] = "10.1145/3592430"` (bare) vs the real
-  `https://doi.org/…` URL form; `test_fetch_paper_*` assert title/year/authors/
-  abstract/`openalex_id` but **never** `paper.id`.
-- **Impact:** the id-mapping — the exact surface of C-7 — is untested, and the
-  fixture actively hides the URL-form behaviour. Any normalization fix needs this
-  closed first or it will regress invisibly.
-
-### R-5 — S2 `fetch_references` breaks on legacy arXiv IDs containing `/` · **LOW**
-- **Evidence:** `semanticscholar.py:80` interpolates `paper_id` into the path
-  `…/paper/{paper_id}/references`. The ingest fallback passes `arXiv:math/0603097`
-  (`ingest.py:91`) — the embedded `/` yields a malformed path → 404 →
-  `fetch_references` silently returns `[]`. Legacy-arXiv papers that miss the
-  OpenAlex path get **zero references** with no error.
-
-### R-4 — `arxiv.fetch_source` no-op try/except + no retry · **LOW**
-- **Evidence:** `arxiv.py:41-44` — `try: httpx.get(...) except httpx.RequestError: raise`
-  catches and immediately re-raises, doing nothing. Unlike OpenAlex/S2, the arXiv
-  client has **no** retry/back-off, so a transient network blip aborts ingest.
-  Minor resilience asymmetry.
-
-### C-9 — `semanticscholar.fetch_recommendations` is unreachable app code · **LOW**
-- **Evidence:** referenced only by `tests/sources/test_semanticscholar.py`; no
-  `codex/` module or CLI calls it (`grep fetch_recommendations`). Tested but
-  never wired — dead feature surface (decide: wire a `discover recommend`
-  command, or drop).
-
-**Part B net:** 1 HIGH (C-7, ID-form contract — the big one; confirm on live DB),
-1 MED (T-3, fixture masks it), 3 LOW (R-4, R-5, C-9). The retry/timeout/rate-limit
-hygiene is genuinely good; the weakness is the **un-normalized, heterogeneous ID
-space** (URL vs bare, W-ID vs DOI vs arXiv vs S2-hash) threaded through `papers.id`
-and `citations.cited_id` — the same root cause behind C-1 and the F-15 ID dance.
-
----
-
-## Part C — `parsing/` (tex, grobid, nougat, mathpix, figures)
-
-> **Scope reality:** the live corpus is ingested entirely as **`.txt`**
-> (`ingest_all.sh` maps every paper to a `*.txt` file), so the `.tex`-clean,
-> GROBID, Nougat, Mathpix, and Figures paths are **latent** — only
-> `tex.chunk_text` + `quality.*` run on the live data. The findings below are
-> real but mostly fire only once `.tex`/`.pdf --rich` ingest is used.
->
-> **Not a finding:** the broad `except Exception` in `mathpix.py:109,151` and
-> `figures.py:125,152` are *justified* per-item graceful degradation
-> (`fitz.open` fails → return empty; pix2tex/image fails → skip that item), not
-> swallowed bugs.
-
-### C-10 — Formulas/figures key on the PDF *filename*, not `paper.id` · **MED** (latent, `.pdf --rich`)
-- **Evidence:** `figures.py:119` and the mathpix extractor set
-  `paper_id = Path(pdf_path).stem`. `ingest.py:289,303` insert
-  `f.paper_id` / `fig.paper_id` **unchanged** into `formulas`/`figures`, whose
-  schema declares `paper_id TEXT REFERENCES papers(id)` (`schema.sql:97,115`).
-- **Impact:** the filename stem (e.g. `springborn-2008-weighted-delaunay…`) will
-  not equal `papers.id` (an OpenAlex URL/bare id — see C-7), so the FK insert
-  **fails and aborts rich ingest**, or (without FK enforcement) writes orphaned
-  rows. Identity is derived from the *file*, not the *paper*. Fix: pass `paper.id`
-  into the extractors, or have ingest overwrite `f.paper_id = paper.id` before insert.
-
-### R-7 — `_clean_latex` misses `\(…\)` / `\[…\]` math + strips all math from chunks · **LOW** (`.tex` path)
-- **Evidence:** `tex.py:21-56` strips `%comment`, `$…$`, `$$…$$`,
-  `equation`/`align` envs, `\cite/\label/\ref` — but **not** the `\(…\)` / `\[…\]`
-  math delimiters. Papers using those leave raw `\( … \)` markup in the chunk text.
-- **Impact:** (a) leftover math markup pollutes embeddings; (b) *all* math is
-  removed from chunk text, so chunk-level semantic search can never match formula
-  content — it relies entirely on the separate formula FTS (F-09), which is itself
-  latent here. Acceptable as a separation-of-concerns choice, but worth stating.
-
-### R-6 — PDF-path external calls are unwrapped / un-retried · **LOW** (`.pdf` path)
-- **Evidence:** `ingest.py:185` calls `extract_references()` outside any
-  `try`; GROBID (`grobid.py:54,133`) and the mathpix HTTP calls `raise_for_status`
-  with **no** retry/back-off (unlike OpenAlex/S2). A transient GROBID/Mathpix
-  5xx aborts the whole PDF ingest.
-- **INFO:** `grobid.py:56` uses stdlib `ET.fromstring` on the server's XML.
-  Fine for a trusted self-hosted GROBID; switch to `defusedxml` if it is ever
-  pointed at an untrusted endpoint.
-
-**Part C net:** 1 MED (C-10), 2 LOW (R-6, R-7), all **latent** for the current
-`.txt` corpus. `nougat.py` was skimmed only (54 lines, `.pdf`-only wrapper) — no
-deep review yet; flagged in the ledger. Parsing hygiene is otherwise good.
-
-> **Request (a) — second audit pass — is now complete:** §8 of the loop-1 doc
-> (grounding guard ✓ Part A, `sources/` ✓ Part B, `parsing/` ✓ Part C) is closed.
-> Remaining whole-repo parts D–G are the broader sweep.
-
----
-
-## Part D — `synthesis.py` orchestration
-
-The lead pipeline (connections → gaps → improvements → conjectures) reuses the
-Part-A grounding guard for the three grounded kinds and degrades gracefully on
-LLM/DB failure (`_safe_llm_generate`/`_list_paper_titles` swallow + log). The
-conjecture invariants are genuinely solid (status hard-set `"unverified"`,
-mandatory falsification path or the lead is dropped, written to `conjectures/`
-only). Two real defects and a doc contradiction:
-
-### C-11 — Lead-ID collision silently overwrites synthesis output · **HIGH**
-- **Evidence:** every stage restarts its counter at `seq = 1`
-  (`synthesis.py:341` connections, `:415` gaps, `:530` improvements,
-  `:634` conjectures) and ids are `L-0001, L-0002, …` (`_make_lead_id`). The CLI
-  concatenates the grounded kinds — `all_leads = connections + gaps + improvements`
-  (`cli.py:421`) — and `write_leads` routes **all** grounded kinds to the *same*
-  `grounded/<id>.md` (`synthesis.py:783-796`), writing in list order.
-- **Bug:** connection `L-0001`, gap `L-0001`, improvement `L-0001` all target
-  `grounded/L-0001.md`; later writes overwrite earlier → **improvements clobber
-  gaps clobber connections** at each number. Surviving files =
-  `max(#connections, #gaps, #improvements)`, **not** the sum. With 5/3/4 leads you
-  get 5 files, not 12; connections are lost first. Silent data loss of the
-  synthesis engine's core output, masked because "files do get written".
-- **Fix direction:** namespace ids by kind (`L-C-0001` / `L-G-0001` / `L-I-0001`)
-  or use a single shared counter across stages.
-
-### R-8 — Default-topic gap detection is a false-positive generator · **MED**
-- **Evidence:** `find_gaps` defaults `probe_topics` to paper **titles** when
-  `topics is None` (`synthesis.py:418-420`); a title query retrieves mostly its
-  *own* paper's chunks → `bibkeys_for_topic` ≈ 1, which is `< synthesis_gap_min_coverage`
-  (`:450-453`) → the topic is flagged as a gap and the LLM is asked to articulate
-  one.
-- **Impact:** nearly every ingested paper becomes a spurious "Gap: '<title>'
-  covered by only 1 bibkey(s)" candidate — the LLM invents a gap for material that
-  is actually well covered. The grounding guard catches some, but the default
-  signal is mostly noise. (User-supplied `--topics` avoid this.)
-
-### D-4 — `write_leads` vs `_update_index` docstrings contradict each other · **LOW**
-- **Evidence:** `write_leads` (`synthesis.py:762-765`) claims it "maintains an
-  **append-only** `INDEX.md`: a previously-recorded id is kept on subsequent runs",
-  but `_update_index` (`:802-803`) states — and implements — a "**full rebuild**,
-  not append-only" by globbing the on-disk dirs.
-- **Impact:** there is no append-only / id-retention logic; the index purely
-  reflects current disk state. The guarantee in the `write_leads` docstring is
-  false (and interacts with C-11: overwritten ids simply vanish from the index).
-
-**Part D net:** 1 HIGH (C-11, silent lead overwrite), 1 MED (R-8, gap noise),
-1 LOW (D-4, doc contradiction). The conjecture path is exemplary; the grounded
-path's persistence layer is where it breaks.
-
----
-
-## Part E — `wiki.py` rest (compile, cross-refs, state, index/log)
-
-Good: `yaml.safe_load`, deterministic `_chunk_hash` (sorted by chunk id) driving a
-sound incremental-skip, append-only `log.md`, inline-code protection during
-cross-ref injection. The issues are math-corruption, a dead step, and a marking
-order bug.
-
-### R-9 — Cross-ref injection corrupts LaTeX math · **MED**
-- **Evidence:** `_inject_cross_refs` (`wiki.py:426`) protects only inline-code
-  spans (`_INLINE_CODE_RE`); it then runs whole-word, case-insensitive title/alias
-  → `[[slug]]` replacement over everything else (`:429-438`). **Math spans
-  (`$…$`, `$$…$$`, `\(…\)`) are not protected.**
-- **Impact:** the synthesis prompt explicitly asks for LaTeX `$ … $`
-  (`wiki.py:495`), so bodies contain math. A concept title/alias that appears
-  inside a formula (e.g. a concept named "Energy" inside `$E_{\text{Energy}}$`,
-  or a single-word alias) is rewritten to `[[slug]]`, corrupting the LaTeX. The
-  risk scales with how common the concept titles/aliases are as math tokens.
-
-### C-12 — `_try_embed_formulas` is a no-op that still costs a DB round-trip per concept · **LOW**
-- **Evidence:** `wiki.py:623-637` opens a connection, queries
-  `information_schema.tables` for `formulas`, and returns — the actual embed is
-  `"(Future: …)"`. Called unconditionally per concept (`compile_concept:609`).
-- **Impact:** step 4 of `compile_concept`'s docstring ("Embed formula chunks")
-  does nothing, and each concept compile pays an extra DB connection + query for
-  no effect. Placeholder masquerading as a feature.
-
-### C-13 — Ungrounded-claim marking can miss claims mutated by cross-ref injection · **LOW**
-- **Evidence:** order in `compile_concept` — claims parsed from `raw_output`
-  (`:602-603`), then `raw_output` is rewritten by `_inject_cross_refs` (`:606`),
-  then `_render_page_markdown` (`:612`) re-finds the *original* `claim.text` by
-  regex in the **injected** body (`:544-552`).
-- **Impact:** if an ungrounded claim's text contained a term that got replaced
-  with `[[slug]]`, the original text no longer matches → the `⚠` mark is silently
-  not applied. (Also note: wiki's marking lacks the `(?<!⚠ )` guard that
-  synthesis's `_mark_ungrounded` has — two divergent mark implementations.)
-
-### R-10 — Non-atomic compile-state write · **LOW**
-- **Evidence:** `_save_compile_state` (`wiki.py:475-477`) writes
-  `.compile-state.json` in place. A crash mid-write corrupts it; `_load`
-  then catches `JSONDecodeError` → `{}` → every concept recompiles. Graceful but
-  loses incrementality. Write to a temp file + `os.replace` for atomicity.
-
-### R-11 — `load_concepts` raises `KeyError` on malformed YAML · **LOW**
-- **Evidence:** `entry["slug"]` / `entry["title"]` (`wiki.py:135-136`) are
-  unguarded; a concepts.yaml entry missing a key crashes the whole compile with a
-  bare `KeyError` rather than a actionable message.
-
-> **Minor:** `compile_all` builds the index from `state.keys()` (`:745`), which
-> accumulates slugs across runs — a concept removed from `concepts.yaml` lingers
-> in `index.md`/state until manually pruned. Low.
-
-**Part E net:** 1 MED (R-9, math corruption), 4 LOW (C-12, C-13, R-10, R-11). The
-incremental-compile machinery is sound; the rendering/cross-ref layer has the rough
-edges. `OllamaClient.generate` (no retry, 120 s timeout, `raise` on error) feeds
-directly into **C-4** (its exception → empty page → published), so C-4's blast
-radius includes every transient Ollama hiccup.
-
----
-
-## Part F — `embed.py`, `quality.py`, `models.py`
-
-The best-built corner of the repo. `embed.py` L2-normalises correctly (cosine ⇒
-dot product), guards empty input and zero-norm rows, and caches a process
-singleton. `quality.py` thresholds are all config-driven, `run_quality_pass` is
-CLI-wired (`cli.py:513-517`) and atomic (single commit). `models.py` dataclasses
-mirror the schema cleanly. Two dead/low-signal items and one corroboration:
-
-### C-14 — The sparse-embedding ("hybrid") path is dead code · **LOW**
-- **Evidence:** `Embedder.encode_sparse` / `encode` / `_coerce_sparse` are called
-  nowhere in `codex/` (grep for `encode_sparse` / `.encode(` finds only
-  `str.encode` in `wiki.py:455`). Only `encode_dense` is wired (ingest, search,
-  wiki). There is no sparse column in the schema.
-- **Impact:** the `embed.py` header and ADR-0002 advertise "**Hybrid dense +
-  sparse** embeddings via BGE-M3" as a headline, but the sparse half is
-  unreachable — the actual "hybrid" elsewhere is dense + Postgres FTS. Either wire
-  sparse retrieval or drop the capability + correct the docs.
-
-### R-12 — `classify_section` is low-signal as fed · **LOW**
-- **Evidence:** `classify_section` matches section headers in the **first 200
-  chars** (`quality.py:103-106`), but `tex.chunk_text` produces overlapping
-  *word-window* chunks that almost never begin at a header. So most chunks fall
-  through to `"body"` (or the DOI-density `"bibliography"` fallback).
-- **Impact:** the `chunks.section` column is mostly `"body"` — limited value for
-  any section-aware retrieval that depends on it. Not wrong, just weak signal
-  given the chunker.
-
-### Corroboration of C-7 (not a new finding)
-- `models.Paper.id` docstring (`models.py:20-21`) states the canonical id is "an
-  arXiv ID … **or a DOI (e.g. `10.1145/3592430`)**" — the **bare** form. Production
-  fills it from OpenAlex in **URL** form (C-7), so the code violates its own model
-  contract. This makes C-7 a documented-contract breach, not a judgement call.
-
-**Part F net:** 2 LOW (C-14, R-12) + a C-7 corroboration. These three modules are
-clean; no correctness defects.
-
----
-
-## Part G — `cli.py` rest + `config.py`
-
-Most CLI commands are thin Typer wrappers delegating to already-audited modules
-(`ingest`, `search`, `discover`, `wiki`, `synthesis`, `provenance`, `quality`,
-`graph`). Settings are pydantic-validated with sensible bounds (`gt=0`, `0..1`).
-Findings are config-coupling and output hygiene.
-
-### C-15 — `embedding_dim` is a footgun decoupled from the hardcoded schema · **MED**
-- **Evidence:** `embedding_dim` is configurable (`config.py:67-74`, default 1024)
-  and `.env.example`/`schema.sql:7` invite other models (Qwen3 1024, **Jina v4
-  2048**). But `schema.sql` hardcodes `vector(1024)` on every embedding column
-  (`:25,38,102,119`), and `Embedder` derives the real output dim from the *model*,
-  using the `dim` setting only for empty-input zero-vectors (`embed.py:50,89`).
-- **Impact:** set `EMBEDDING_DIM=2048` (or pick a non-1024 model) and inserts
-  fail on a pgvector dimension mismatch — silently for empty abstracts
-  (`(1,2048)` zero-vector into `vector(1024)`), structurally for real vectors.
-  Nothing couples the knob to the DDL; the setting reads as supported but is not.
-
-### R-13 — `export_bib` does no BibTeX escaping and forces `@article` · **LOW**
-- **Evidence:** `provenance.py:174-181` interpolates `title`/`author`/`year` raw
-  into `@article{…}`; no escaping/brace-balancing of BibTeX specials
-  (`{ } \ % # $ _ &`).
-- **Impact:** a title with an unbalanced brace or a stray special char produces a
-  malformed `.bib` entry that breaks Doxygen/BibTeX. Also every paper is emitted
-  as `@article` although the corpus includes theses/books/chapters
-  (`ingest_all.sh`) — bibliographically wrong (functional for `@cite` keys though).
-
-### S-4 — Secrets stored as plain `str`, not `SecretStr` · **LOW**
-- **Evidence:** `database_url`, `mathpix_app_key`, `mcp_auth_token`
-  (`config.py:29,148,249`) are plain `str`. Any `repr(Settings())` or debug log of
-  the settings object prints the DB password / API key / bearer token in clear.
-- **Impact:** log/exception-dump leakage surface (complements S-1). Use
-  `pydantic.SecretStr` for credential fields.
-
-### R-14 — `mcp_transport` enum not enforced · **LOW**
-- **Evidence:** `config.py:229-236` documents "stdio or http" but no validator;
-  `main()` (`mcp_server.py:323`) treats anything non-`http` as stdio. A typo
-  (`htttp`) silently starts stdio instead of failing loudly.
-
-### D-5 — Inconsistent `validation_alias` on `nougat_url` only · **LOW**
-- **Evidence:** `nougat_url` adds `AliasChoices("NOUGAT_URL", "nougat_url")`
-  (`config.py:47`) that no other field has — redundant under
-  `case_sensitive=False`, and the inconsistency invites confusion.
-
-> **Corroboration of C-7:** `cli.py:45` ingest does **no** id normalization and
-> echoes `result.paper_id` (the URL-form id) straight back — no normalization
-> layer exists at any tier (source → ingest → CLI).
-
-**Part G net:** 1 MED (C-15, dim/schema footgun), 4 LOW (R-13, S-4, R-14, D-5).
-CLI/config are otherwise solid and well-documented.
-
----
-
-# Executive Summary — whole-repo audit complete (loop-1 + Parts A–G)
-
-**Tally: 45 findings — 8 HIGH, 11 MED, 26 LOW/INFO** (after live-DB verification:
-**7 HIGH, 12 MED** — M-1 reclassified, see the verification section at the bottom).
-Test suite 330 green, `ruff`/`mypy` clean. The codebase is well-structured and idiomatic; the defects
-cluster into five root themes, not random scatter.
-
-### The 8 HIGH findings
-| ID | Theme | One-liner |
-|----|-------|-----------|
-| S-1 | secrets | live DB password in un-ignored `.env.jetson-ingest` (mitigated this session) |
-| M-1 | migration | `paper_identifiers` has no migration path; `schema.sql` not re-applyable, `apply_schema` never called |
-| C-1 | identity | `discover.py` keeps the ID-mismatch bug F-15 fixed in `graph.py` (over-reports ingested papers as leads) |
-| C-7 | identity | `papers.id` stored in OpenAlex **URL** form, violating the bare-ID contract (breaks lookups + cross-citation match) |
-| C-4 | silent failure | zero-citation/LLM-outage wiki pages **bypass quarantine** and publish as "compiled" |
-| C-11 | silent data loss | synthesis leads collide on `L-000N` across stages → improvements overwrite gaps overwrite connections |
-| D-1 | doc integrity | ADR-F15 GO rests on spike numbers measured **before** the shipped resolution fix |
-| T-1 | verification | the F-15 resolution SQL (the substance of the last 2 commits) has **zero** real-DB coverage |
-
-### Five root-cause themes
-1. **Un-normalized identity (the dominant theme).** C-1, C-7, C-10, M-1, T-3,
-   and most of the F-15 ID dance are one problem: a paper's identity exists in
-   many un-reconciled forms (bare vs URL · DOI vs arXiv vs OpenAlex-W vs S2-hash
-   vs PDF-filename) with **no canonical normalization layer**. One normalization
-   boundary at ingest would retire ~5 HIGH/MED findings at once.
-2. **"Green but unverified."** T-1/T-2/T-3 — the most error-prone code (resolution
-   SQL, migrations, id-mapping) is mocked or tested with unrepresentative fixtures,
-   so real bugs (C-4, C-7, C-11) sit behind 330 passing tests.
-3. **Silent failure / data loss under degradation.** C-4 (publish empty), C-11
-   (overwrite leads), R-1 (over-quarantine) corrupt *outputs* instead of erroring
-   — the worst failure mode for a research tool whose value is trust.
-4. **Overstated guarantees.** D-1 (stale ADR), D-2/D-4 (doc↔code drift), C-5
-   ("zero-fact-leak" ≫ a last-sentence substring check), C-14 ("hybrid" sparse is
-   dead) — the docs/commit-language claim more than the code delivers.
-5. **Latent rich-path debt.** C-10, R-6, R-7 are real but dormant because the live
-   corpus is `.txt`; they detonate on the first `.pdf --rich` ingest.
-
-### Suggested remediation sequencing (themes, not a plan — planning deferred)
-- **First:** S-1 (rotate credential — secret already protected), M-1 (migration
-  path), C-7+C-1 (one identity-normalization layer), T-1 (real-DB test) — these
-  unblock trust in the live corpus.
-- **Then:** C-4 + C-11 (stop silent output corruption), D-1 (re-validate ADR-F15).
-- **Then:** MED batch (R-1, R-8, R-9, C-15, C-10) and the LOW polish.
-
-### Coverage honesty
-Every `codex/` module was read. After the follow-up pass **all are deep-reviewed**
-(`nougat.py` included — see Runtime Validation). Runtime state that was initially
-*inferred from code* (C-7 id-form, M-1 table existence, and the C-4/C-11
-behaviours) has since been **empirically confirmed** — see the *Live DB
-Verification* and *Runtime Validation* sections below.
-
----
-
-# Live DB Verification — 2026-06-15 (read-only, via existing SSH tunnel)
-
-Ran read-only queries against the live Jetson corpus (`papers=36`,
-`citations=811`). This **confirms the two identity HIGHs with hard numbers** and
-**corrects an over-statement in M-1**.
-
-### C-7 — CONFIRMED ✅ (severity stands: HIGH)
-`papers.id` form distribution: **`doi_url=35`, `bare_arxiv=1`, everything else 0**
-(of 36). Samples: `https://doi.org/10.48550/arxiv.math/0603097`,
-`https://doi.org/10.48550/arxiv.2310.17529`. Even arXiv papers are stored as their
-arXiv-DOI **URL**, never the bare `math/0603097` / `2301.x` the schema, `models.Paper`,
-and `ingest_all.sh` assume. The single bare id is the one paper OpenAlex 404'd
-(fell to `ingest.py:94`). → bare-ID lookups fail for 35/36 papers. Confirmed.
-
-### C-1 — CONFIRMED ✅ with impact (severity stands: HIGH)
-- `citations.cited_id` is uniformly **`oa_url=811`** (all OpenAlex URLs) — the live
-  corpus was ingested via the OpenAlex path, so the heterogeneous bare-DOI/S2 mix
-  theorised in Part B is **not present here** (it would appear only via S2/GROBID,
-  i.e. `.pdf`/fallback ingest).
-- **`discover.py` wrongly flags 13 already-ingested papers as "dangling / not
-  ingested"** (their `cited_id` is an OpenAlex URL whose paper *is* in the corpus
-  via `openalex_id`). Meanwhile **`graph.py` correctly resolves 57 cross-citation
-  edges** via the `openalex_id` JOIN. The two "dangling" implementations
-  measurably disagree — exactly the C-1 divergence, now quantified.
-
-### M-1 — REFINED ⤵ (downgrade acute severity; structural finding stands)
-`paper_identifiers` **exists on the live DB with 39 rows** — the catastrophic
-"graph JOIN crashes with relation-does-not-exist" scenario **does not occur**;
-the table was applied (manually or via DB re-create). The *structural* gap stands:
-`apply_schema` is still non-idempotent and never invoked, so the **next** schema
-change has no migration path — but the acute "live graph is broken" framing was
-**over-stated**. Re-rank M-1 from HIGH → **MED** (latent migration debt, not an
-active outage).
-
-### C-2 — PARTIALLY REFUTED ⤵
-`paper_identifiers` has **39 rows vs ~35 papers-with-openalex_id**, so ~4 papers
-carry a diverging alias → the `pi` JOIN is **not** fully inert (it resolves a
-handful of cases the `p` JOIN can't). Keep as LOW/INFO: the table earns its keep
-for a few papers, though the cost/benefit at this corpus size is still marginal.
-
-### Net adjustment to the tally
-HIGH **8 → 7** (M-1 → MED). C-7 and C-1 are now **confirmed**, not inferred — they
-should lead remediation. The dominant root theme (**un-normalized identity**) is
-empirically validated: 35/36 ids in the "wrong" form, 13 papers mis-classified by
-the older code path, 57 edges rescued by the newer one.
-
----
-
-# Runtime Validation — 2026-06-15 (C-4 and C-11 reproduced end-to-end)
-
-The two HIGH findings flagged "code-certain, runtime-confirmable" were exercised
-against the **real** code paths (injected fake LLM; no Ollama, no DB needed).
-
-### C-11 — CONFIRMED ✅ (lead-ID collision → silent loss; stays HIGH)
-Reproduced the exact CLI assembly (`cli.py:421` `connections + gaps +
-improvements`, each stage numbered from `L-0001`): **4 distinct leads** (2
-connection, 1 gap, 1 improvement) → `write_leads` produced **2 files**.
-`grounded/L-0001.md` survived as the **improvement** ("IMPROVEMENT-one"); the
-connection and gap `L-0001` were silently overwritten. **2 of 4 leads lost.**
-
-### C-4 — CONFIRMED ✅ (zero-claim page bypasses quarantine; stays HIGH)
-Drove the real `compile_all` with a fake LLM returning confident **uncited** prose
-(→ 0 parseable claims). With `wiki_min_grounding_rate = 0.5` and an actual
-grounding rate of `0.0`, the page was **published to `wiki/` and marked
-compiled** (`report.compiled=['test-concept']`, `report.quarantined=[]`) — not
-quarantined to `wiki/draft/`. The `total_claims > 0` guard (`wiki.py:730`) is the
-exact bypass; the bug fires precisely when grounding evidence is weakest.
-
-### `nougat.py` — now fully reviewed (no new finding)
-Folds into **R-6** (PDF-path resilience). Detail: `pdf_to_markdown` retries **only**
-`httpx.ConnectError` with `wait_fixed(0)` (no 5xx retry, no back-off) and is
-unwrapped at `ingest.py:183` — a Nougat 5xx aborts PDF ingest.
-
-**Closure:** every `codex/` module is deep-reviewed; the two runtime-confirmable
-HIGHs are empirically reproduced. The **audit / finding-identification pass is
-complete**. Remediation planning is the open next phase.
-
----
-
-# Migration Incident — 2026-06-15 (M-1 confirmed live, new privilege dimension)
-
-While running the C-7 re-ingest migration (`infra/reingest_canonical_ids.sh`),
-M-1 manifested exactly as predicted — and exposed a dimension the static audit
-did **not** capture.
-
-**What happened.** The helper ran `TRUNCATE papers CASCADE` (corpus wiped to 0),
-then `ingest_all.sh` failed on the first chunk insert:
-`UndefinedColumn: column "section" of relation "chunks" does not exist`. The live
-`chunks` table predates the F-16 `section` column, and nothing had ever applied
-that DDL to the live DB (M-1: `apply_schema` is never called; `schema.sql` is not
-re-applyable). The ingest code was ahead of the live schema.
-
-**New finding — privilege separation (extends M-1).** The fix
-`ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT` then failed with
-`InsufficientPrivilege: must be owner of table chunks`. The live ownership:
-
-| table | owner |
-|-------|-------|
-| papers, chunks, citations, code_links, formulas, figures | `postgres` |
-| paper_identifiers | `researcher` (app user) |
-
-The app role `researcher` has DML + `TRUNCATE` but **not** DDL on the
-postgres-owned core tables. So a `codex migrate` / `apply_schema` invoked with the
-application's `DATABASE_URL` would fail on every `ALTER` / `CREATE INDEX` against
-those tables. **Implication for the M-1 fix:** making `schema.sql` idempotent is
-necessary but *not sufficient* — the migration must run under a privileged role
-(owner/superuser), via a separate admin connection, not the least-privilege app
-user. This is now part of M-1's remediation scope.
-
-**Consequences observed.**
-- Corpus left wiped (papers = 0) until the owner adds the column and re-ingest is
-  re-run — a destructive-then-fail sequence.
-- Helper hardened (`fix(infra)` on `fix/audit-wave-2`): a **preflight** verifies
-  `chunks.section` exists *before* the `TRUNCATE`, aborting early with the
-  owner-level `ALTER` to run, so a schema gap can no longer cost the corpus.
-
-**Unblock (owner action):**
-`sudo -u postgres psql -d papers -c "ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT;"`
-then re-run `bash ingest_all.sh`.
-
-This incident is the strongest possible validation of M-1 and should anchor its
-Wave-3 fix: **idempotent `schema.sql` + a `codex migrate` command that connects as
-a privileged role**, plus a documented owner/grant model for the app user.

docs/audit/AUDIT-2026-06-15-loop-1.md

@@ -1,227 +0,0 @@
-# AUDIT — Development Loop 1 (main branch)
-
-**Auditor:** Audit-Loop (Opus)
-**Date:** 2026-06-15
-**Scope:** entire `main` branch (HEAD `5dfa6e4`)
-**Method:** static review of source + schema + tests, branch-wide pattern scan
-(`ruff`, `mypy`, defect-class greps), full test run (`330 passed`).
-
-> Status legend — **HIGH** = fix before relying on the feature ·
-> **MED** = fix soon, correctness/maintainability · **LOW** = polish ·
-> **INFO** = accepted risk / no action, documented for the record.
-
-This document **only records findings**. Remediation planning is a separate
-step (next phase of the audit loop).
-
----
-
-## 0. Headline
-
-The most recent dev-loop work (**F-15 citation graph** + the follow-up fixes
-`cited_id→papers.id` resolution and the `paper_identifiers` alias table) is
-**functionally green but unverified where it matters**: the SQL that *is* the
-fix has zero real-database coverage, the live-corpus validation in the ADR
-predates the fix, and the same ID-mismatch bug the fix solves still lives in a
-parallel code path (`discover.py`). Separately, a **live DB password sits in an
-un-ignored file** and the new table has **no migration path onto the live DB**.
-
-Test suite: **330 passed**, `ruff check` clean, `ruff format --check` clean,
-`mypy` clean on reviewed modules.
-
----
-
-## 1. Security
-
-### S-1 — Live DB credential in an un-ignored file · **HIGH**
-- **Evidence:** `.env.jetson-ingest` contains
-  `DATABASE_URL=postgresql://researcher:05761b87…@localhost:5433/papers`.
-  `.gitignore` ignores only the literal `.env` (line: `# Environment secrets — NEVER commit` → `.env`).
-  `git check-ignore .env.jetson-ingest` → **NOT IGNORED**.
-- **Impact:** one `git add -A && git commit` leaks a real Postgres password to
-  history. The file is currently untracked (`??`), so the leak has **not**
-  happened yet — this is a near-miss, not an incident.
-- **Note:** the same gap exposes any future `.env.<profile>` file.
-
-### S-2 — Non-constant-time token comparison (MCP HTTP) · **LOW**
-- **Evidence:** `codex/mcp_server.py:340` — `if auth != f"Bearer {token}":`.
-- **Impact:** theoretical timing side-channel on the Bearer token. Low on a
-  trusted LAN, but trivially fixable with `secrets.compare_digest`.
-
-### S-3 — DNS-rebinding protection disabled · **INFO (accepted)**
-- **Evidence:** `codex/mcp_server.py:35-37`,
-  `TransportSecuritySettings(enable_dns_rebinding_protection=False)` (commit `d29dcf2`).
-- **Assessment:** documented and mitigated — HTTP transport requires a Bearer
-  token (`_TokenAuth`) and is firewalled to LAN IPs (UFW). A rebound browser
-  request lacks the token. Residual risk only if the firewall assumption fails.
-  No action; recorded so the trade-off stays visible.
-
----
-
-## 2. Correctness
-
-### C-1 — `discover.py` still has the ID-format-mismatch bug F-15 fixed · **HIGH**
-- **Evidence:** `codex/discover.py:20` —
-  `WHERE cited_id NOT IN (SELECT id FROM papers)`; same pattern in
-  `cocited_papers` (`discover.py:52-61`), `citing_papers`, `cited_by`.
-- **Root cause:** `citations.cited_id` is a **mix** of OpenAlex W-IDs (OpenAlex
-  API path, `ingest.py:229-231` ← `sources/openalex.py:155`) and DOIs/arXiv
-  (GROBID/S2 paths). `papers.id` is the canonical DOI/arXiv. An ingested paper
-  cited by its **OpenAlex ID** is *not* in `(SELECT id FROM papers)`, so it is
-  wrongly reported as a discovery lead.
-- **Impact:** `codex discover leads`, `codex discover cocited`, and the MCP
-  `discover_leads` tool over-report — foundational papers the user has **already
-  ingested** show up as "cited but not ingested". This is exactly the
-  D-05-class issue the F-15 graph path resolves via the
-  `papers.openalex_id` JOIN (`graph.py:48-56`) — but the fix was **not
-  propagated** to `discover.py`. Two divergent "dangling" implementations now
-  disagree on what "ingested" means.
-
-### C-2 — `paper_identifiers` JOIN is effectively inert · **MED**
-- **Evidence:** `graph.py:50-56` second LEFT JOIN; populated only at
-  `ingest.py:153-161`, which inserts **the same** `paper.openalex_id` that
-  `papers.openalex_id` already holds.
-- **Impact:** because ingest never records an alias that differs from
-  `papers.openalex_id`, the `pi` JOIN can only resolve what the `p` JOIN already
-  resolves. It adds value **only** in the narrow case where a paper's
-  `openalex_id` changes across re-ingests (old alias retained via
-  `ON CONFLICT DO NOTHING`). The table + unique index + seed INSERT + migration
-  burden buy near-zero behaviour today — speculative generality. No test
-  exercises an alias that diverges from `papers.openalex_id`.
-
-### C-3 — MCP `wiki_read` returns the slug as its own "sources" · **MED**
-- **Evidence:** `codex/mcp_server.py:138-149` — comment says "Collect cited
-  bibkeys", code sets `sources = [concept_slug]`.
-- **Impact:** the `sources` field is a placeholder masquerading as data; any MCP
-  client trusting it to list cited bibkeys gets the concept's own slug.
-
----
-
-## 3. Documentation / Code Drift
-
-### D-1 — ADR-F15 validation predates the shipped graph · **HIGH (integrity)**
-- **Evidence:** ADR-F15 (commit `a1a6f45`) lands **before** the resolution fixes
-  `664906f` (`cited_id→papers.id`) and `5dfa6e4` (`paper_identifiers`).
-- **Stale claims:**
-  - "Spike Result": *478 dangling nodes*, specific PageRank scores,
-    *Bobenko/Springborn rank 7*, *5.7 % spread* — all measured on the
-    **pre-resolution** (bipartite DOI→OpenAlex) graph. Post-fix, in-KB cited
-    papers resolve to DOIs, so dangling count and PageRank distribution change.
-  - "Decision → Graph Construction": "inter-KB citations are structurally
-    invisible" — **now false**.
-  - "Known Limitation: ID-Format Mismatch": "cross-citations… not represented"
-    — **now resolved by the code**, contradicting the section.
-- **Impact:** the **GO** decision rests on numbers the shipping code no longer
-  produces. Validation must be re-run on the post-fix graph (or the ADR marked
-  validation-pending).
-
-### D-2 — `graph_cite_boost_alpha` doc describes the *fixed* bug · **MED**
-- **Evidence:** `config.py:296-300` describes
-  `Final score = dense_score * (1 + alpha * pagerank_score)` — a **multiply**.
-  Actual code (`cli.py:107`) divides distance:
-  `boosted = distance / (1.0 + alpha * pr_score)`. The ADR explicitly states the
-  multiply form was the "critical bug corrected during the F-15 review gate".
-- **Impact:** the config docstring still documents the inverted (buggy) formula.
-
-### D-3 — `paper_identifiers` tagged "F-17" in schema, shipped as F-15 fix · **LOW**
-- **Evidence:** `infra/schema.sql:129` comment "F-17 Paper identifier aliases",
-  but the table is introduced by the F-15 graph follow-up (`5dfa6e4`) and the
-  ADR-F15 work. Feature-tag inconsistency; ADR-F15 doesn't mention the table at all.
-
----
-
-## 4. Schema / Migration
-
-### M-1 — New table has no migration path onto the live DB · **HIGH (operational)**
-- **Evidence chain:**
-  1. `apply_schema` docstring (`db.py:46-49`) claims "all DDL statements use
-     CREATE … IF NOT EXISTS" — **false**: `papers`, `chunks`, `citations`,
-     `code_links` and their indexes use bare `CREATE TABLE/INDEX`
-     (`schema.sql:16,33,56,68,30,41,…`). Only F-09/F-16/F-17 DDL is idempotent.
-  2. `apply_schema` is **never called** anywhere in the app — no `init`/`migrate`
-     CLI command (`grep apply_schema` → only `db.py` + a mocked unit test).
-  3. `ingest_all.sh` does **not** apply the schema; it only runs `codex ingest`.
-- **Impact:** re-running `infra/schema.sql` against the **existing** Jetson DB
-  fails on the first `CREATE TABLE papers` (already exists) and rolls back the
-  whole script in one implicit transaction — so the new `paper_identifiers`
-  table (and its seed INSERT) is **never created** on a DB that predates it.
-  `build_citation_graph` then fails with `relation "paper_identifiers" does not
-  exist` on the live corpus. The migration step is currently manual and undocumented.
-
----
-
-## 5. Test Integrity
-
-### T-1 — The F-15 resolution SQL has zero real coverage · **HIGH**
-- **Evidence:** every test in `tests/graph/test_graph.py` and
-  `tests/graph/test_cli.py` builds the graph from a `MagicMock` conn
-  (`_make_conn`, `_make_conn_with_paper_ids`). `test_cross_citation_uses_doi_when_in_kb`
-  (`test_graph.py:83-99`) **pre-resolves** the rows in Python and asserts the
-  graph builds — it tests the mock, not the `COALESCE(p.id, pi.paper_id, …)` +
-  two LEFT JOINs that *are* the fix.
-- **Impact:** the substance of the two most recent commits is unverified. 330
-  green tests give false confidence about the resolution logic, the `p.id IS NULL`
-  correlated guard, and the `paper_identifiers` unique-index behaviour. Needs an
-  integration test against a real (or `pytest`-managed) Postgres.
-
-### T-2 — `apply_schema` idempotency untested · **MED**
-- **Evidence:** `tests/scaffold/test_db.py:24-32` mocks the conn and only checks
-  `execute`/`commit` are called. The (false) idempotency claim in M-1 is never
-  exercised.
-
----
-
-## 6. Design / Usability
-
-### U-1 — `graph report` prints bare IDs, no titles · **LOW**
-- `cli.py:582-588` emits `score  <id>` and bare dangling IDs — a mix of DOIs and
-  `W…` OpenAlex IDs with no title/year. The ADR's readable author/title table is
-  not what the CLI produces; output is hard to interpret.
-
-### U-2 — `--cite-boost` can only reorder within the top-`limit` page · **LOW**
-- `cli.py:84-94` applies `ORDER BY … LIMIT limit` in SQL **before** the boost
-  (`cli.py:96-109`). A high-PR paper just outside the semantic cut can never be
-  pulled in. With α=0.3 and PR≈0.002 the effect is ~0.06 % — effectively a
-  within-page tie-breaker. Consistent with ADR intent, but worth stating as a
-  limitation (the flag does far less than "weight results by PageRank" implies).
-
-### U-3 — `graph report --json` drops the small-corpus warning · **LOW**
-- `cli.py:557-570` returns before the `graph_min_corpus_size` warning
-  (`cli.py:572-577`). JSON consumers get neither the warning nor a flag for it.
-
-### U-4 — `graph.find_co_cited` implemented + tested but not wired to a CLI · **LOW**
-- `graph.find_co_cited` (`graph.py:142`) is unit-tested but exposed by no `graph`
-  sub-command (only the separate SQL-based `discover cocited` is wired). ADR-F15
-  R-43's co-citation is only half-surfaced.
-
----
-
-## 7. State of Tree (INFO — no defect)
-
-- The uncommitted diff (`wiki.py`, `synthesis.py`, `semanticscholar.py`,
-  `tests/synthesis/test_cli.py`, `tests/wiki/test_compile.py`) is a **legitimate
-  `ruff format` normalization** to `line-length=100`. `ruff format --check`
-  reports all 25 files formatted *with* these changes applied, meaning the
-  **committed** code was non-compliant. These edits should be **committed**, not
-  reverted. (Not a finding; clarifies the working-tree churn.)
-
----
-
-## 8. Coverage of this audit (honesty ledger)
-
-**Deep-reviewed:** `graph.py`, `db.py`, `infra/schema.sql`, `ingest.py`,
-`discover.py`, `provenance.py` (scan), `mcp_server.py`, `cli.py` (search +
-graph), `config.py` (graph settings), F-15 tests, ADR-F15; branch-wide pattern
-scan (SQL injection, bare except, eval/exec/subprocess, secrets, timeouts).
-
-**Not yet deep-reviewed (no findings asserted):**
-- `synthesis.py` grounding guard / claim parser (`_run_grounding_guard`,
-  `_parse_claims`) — the "zero-fact-leak (N-02)" core. Prompts reviewed; guard
-  logic not yet traced.
-- `wiki.py` grounding guard + conflict detection (only the format diff seen).
-- `sources/` (arxiv, openalex, semanticscholar) beyond citation shapes.
-- `parsing/` (grobid, nougat, mathpix, figures, tex) — error handling in
-  `mathpix.py`/`figures.py` uses broad `except Exception` without `noqa`
-  (`mathpix.py:109,151`, `figures.py:125,152`) — flagged for a later pass.
-- `embed.py`, `quality.py`, `models.py`.
-
-A second pass should close these before the audit is declared complete.

docs/audit/DATA-QUALITY-2026-06-15.md

@@ -1,575 +0,0 @@
-# DATA-QUALITY AUDIT — codex knowledge base (handoff for a fresh session)
-
-**Status:** baseline scan done 2026-06-15. **DQ-1 RESOLVED 2026-06-16** (S2 supplement
-applied to live DB + wired into ingest). **DQ-4 MEASURED 2026-06-16** — retrieval is
-strong after fixing a P0 `codex search paper` crash. **DQ-2 RESOLVED 2026-06-16**
-(1911.00966 fully recovered from arXiv; s00454 abstract from S2; book chapter is a
-documented limit; recovery wired into ingest). **DQ-3 RESOLVED 2026-06-16**
-(content fidelity clean — full coverage, only OCR junk dropped). **All four
-assessment items (DQ-1..DQ-4) DONE. DQ-5 RESOLVED 2026-06-17** (DOI id normalized
-to bare canonical form in `openalex._map_paper`; live re-ingest of
-`10.1007/s00454-019-00132-8` confirmed idempotent → **unblocks R-A/R-C**).
-**Roadmap R-A..R-E** added 2026-06-16 (free-source acquisition levers — see Roadmap section).
-**Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory
-of the audit conversation — everything needed to continue is in this file.
-
----
-
-## Why this exists — the second audit axis
-
-The code/pipeline audit (`AUDIT-2026-06-15-loop-1.md`, `-full-repo.md`) verified the
-**loader is correct**: IDs are canonical, the citation graph is consistent, the
-ingest pipeline does what it claims. It did **not** ask whether the *information
-in the database is actually good*. A correct loader can faithfully load thin,
-incomplete, or unrepresentative data. This document is that second axis — a
-**data-quality** assessment of the live corpus, independent of code correctness.
-
----
-
-## How to reach the data (self-contained)
-
-- **Live DB:** Jetson PostgreSQL via an SSH tunnel (open it first):
-  ```
-  ssh -f -N -L 5433:localhost:5432 alfred@jetson.local
-  ```
-- **Credentials:** `.env.jetson-ingest` (gitignored) holds `DATABASE_URL`
-  (`postgresql://researcher:…@localhost:5433/papers`). The `researcher` role is
-  DML-only (read freely; no DDL).
-- **Run probes:** `set -a; source .env.jetson-ingest; set +a` then
-  `PYTHONPATH=. .venv/bin/python` with `psycopg` + `psycopg.rows.dict_row`.
-  Reusable domain helpers: `codex.quality` (`is_quality_chunk`, `_bib_score`,
-  `classify_section`), `codex.graph`, `codex.discover`.
-- **Corpus snapshot (2026-06-15, post canonical-id migration):**
-  **29 papers, 1507 chunks, 590 citations.**
-
----
-
-## Baseline — what is already GOOD (measured, no action needed)
-
-- **Chunk content is clean.** Re-running the F-16 quality gate over all stored
-  chunks: **0 / 1507 fail** `is_quality_chunk` (none too-short, none low-alpha /
-  OCR-ish, none bibliography-like). The gate did its job on the `.txt` ingest.
-- **Chunk sizing healthy.** chars: min 478 / median 2913 / max 4154; **0 exact
-  duplicate** chunk bodies.
-- **`section` column is low-signal** (98 % `body`; 13 `bibliography`, 9 `theorem`,
-  2 `intro`, 1 `proof`). This confirms code-audit **R-12** — word-window chunks
-  rarely start at a section header. Not a data defect; the column is just weak.
-
----
-
-## Findings (open investigation items)
-
-### DQ-1 — Citation coverage: 12 / 29 papers (41 %) had ZERO citations · **RESOLVED 2026-06-16**
-
-**Resolution (what was done):**
-- **Verified upstream, not a bug.** Queried the OpenAlex API directly for all 11
-  zero-citation papers that have an `openalex_id`: every one returns
-  `referenced_works_count = 0` server-side, W-ids all match what we stored. They
-  are `type: preprint` (arXiv) / `article` / `dissertation` — works OpenAlex
-  indexes without a parsed bibliography. Confirmed source-coverage limit.
-- **Semantic Scholar supplement applied to the live DB** (idempotent backfill,
-  `ON CONFLICT (citing_id,cited_id) DO NOTHING`): **+330 citations (590 → 920)**.
-  Zero-out-edge papers **12 → 2**; in-corpus citing coverage **17/29 → 27/29**;
-  graph 489→704 nodes, 590→920 edges; discovery-lead targets 472→677. 9 of the
-  new edges are in-corpus (e.g. `2305.10988 → 10.1007/s00454-019-00132-8`).
-- **Wired into ingest** so future re-ingests self-heal: when OpenAlex returns an
-  empty `referenced_works`, `ingest.py` now falls back to
-  `_s2_reference_supplement()` (citing_id rewritten to canonical `papers.id`,
-  DOI cited-ids lowercased). Same helper drove the one-off backfill.
-- **Bug fixed (would have crashed the batch):** `semanticscholar.fetch_references`
-  used `data.get("data", [])`, but S2 returns HTTP 200 `{"data": null}` for a
-  known paper with no parsed refs (e.g. `depositonce-5415`) → `TypeError`.
-  Changed to `data.get("data") or []`. Removed a now-redundant discarded S2 probe
-  in the OpenAlex-404 arXiv branch. New regression test added.
-- **Irreducible remainder (documented limit):** the **2 `depositonce` theses**
-  (`10.14279/depositonce-20357`, `-5415`) stay at zero — neither OpenAlex nor S2
-  indexes references for TU-Berlin institutional DOIs. Accept as inherent.
-- **Files touched (uncommitted in working tree):** `codex/sources/semanticscholar.py`,
-  `codex/ingest.py`, `tests/ingest/test_ingest.py`. Full suite: 331 passed.
-
-**Original finding (for context):**
-- **Measured:** 12 papers contribute **no** out-edges; the 590-edge citation graph
-  comes from only ~17 papers. The whole F-15 layer (PageRank / coupling /
-  co-citation / discovery leads) therefore runs on ~59 % of the corpus.
-- **Sharpening — it is mostly a source-coverage limit, not an ingest bug:**
-  **11 of the 12** zero-citation papers *do* have an `openalex_id`, i.e. ingest
-  called `openalex.fetch_citations(openalex_id)` and OpenAlex returned an **empty
-  `referenced_works`** list. Only `1911.00966` has no `openalex_id` (OpenAlex
-  404 → arXiv/S2 fallback, which also yielded nothing).
-- **Zero-citation ids:** `1005.2698`, `1505.01341`, `1911.00966` (no oa),
-  `2206.13461`, `2305.10988`, `2310.17529`, `2601.22903`, `math/0001176`,
-  `math/0503219`, `math/0603097`, `10.14279/depositonce-20357`,
-  `10.14279/depositonce-5415`. (Pattern: arXiv preprints + the two `depositonce`
-  theses — works OpenAlex indexes without parsed references.)
-- **To investigate next:**
-  1. Confirm it is OpenAlex coverage, not a `fetch_citations` bug: for 2–3 of the
-     ids, hit `GET https://api.openalex.org/works/<openalex_id>` and check whether
-     `referenced_works` is genuinely empty server-side.
-  2. If genuinely empty: enrich via the **Semantic Scholar references** path
-     (`semanticscholar.fetch_references("arXiv:<id>")`) as a *supplement* for
-     OpenAlex-empty papers — S2 often has references where OpenAlex doesn't. Note
-     S2 cited_ids are bare DOI/arXiv, so they flow through the existing
-     `RESOLVED_CITATIONS_SQL` resolver (audit C-1) fine.
-  3. Decide whether the F-15 `graph_min_corpus_size` warning should also flag *low
-     citing-paper coverage*, not just paper count.
-- **Acceptance:** either (a) citation coverage materially improves after an S2
-  supplement, or (b) documented as an inherent OpenAlex-coverage limit with the
-  graph caveated accordingly.
-
-### DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · **RESOLVED 2026-06-16**
-
-**Resolution (what was done):**
-- **`1911.00966` fully recovered** from the arXiv Atom API (OpenAlex 404'd on it):
-  title "A discrete version of Liouville's theorem on conformal maps",
-  Pinkall & Springborn, 2019, 384-char abstract → bibkey `PinkallSpringborn2019`
-  auto-generated, abstract embedded (non-zero). It was the worst node in the
-  corpus — now `@cite`-able, in wiki grounding, and its **10 chunks are visible
-  to chunk search** (previously filtered out by `bibkey IS NOT NULL`). Citations
-  were already added in DQ-1 (27 S2 refs).
-- **`10.1007/s00454-019-00132-8` abstract recovered from S2** (1186 chars) and
-  re-embedded (targeted `UPDATE`, not full re-ingest — see DQ-5).
-- **`10.1007/978-3-642-17413-1_7` (book chapter): documented inherent limit.**
-  No abstract in OpenAlex, S2, **or Crossref** (verified). Left as the single
-  remaining zero-vector paper; user may add an abstract by hand.
-- **Corpus now: 1 paper without abstract** (the book chapter), down from 3+1.
-  Zero-vector search pollution (DQ-4 secondary finding) reduced from 2 tail
-  fillers to 1.
-- **Wired into ingest** so this self-heals: OpenAlex-404 on an arXiv id now
-  recovers metadata via `arxiv.fetch_metadata` (new); an empty abstract is
-  supplemented via `_recover_abstract` → S2 then Crossref (new
-  `codex/sources/crossref.py`). New `semanticscholar.fetch_abstract`.
-- **Files touched:** `codex/sources/arxiv.py`, `codex/sources/crossref.py` (new),
-  `codex/sources/semanticscholar.py`, `codex/ingest.py`, + tests. Full suite: 342.
-
-**Original finding (for context):**
-- **Measured:**
-  - **No abstract (3):** `10.1007/978-3-642-17413-1_7`,
-    `10.1007/s00454-019-00132-8`, `1911.00966`. These get a **zero-vector**
-    `abstract_emb`, so paper-level semantic search can't place them.
-  - **No bibkey / year (1):** `1911.00966` only — and its authors array is empty
-    too. This paper is **fully degraded** (OpenAlex 404 → `Paper(id, title="")`
-    fallback): no abstract, no bibkey, no year, empty authors, no citations. With
-    no bibkey it cannot be `@cite`d and is invisible to wiki grounding (which keys
-    on bibkey).
-- **To investigate next:**
-  1. `1911.00966` — confirm the arXiv id is correct and whether OpenAlex/S2 has it
-     under a different id (DOI?); if recoverable, re-ingest to populate metadata.
-     If not, decide: keep as a degraded node or drop.
-  2. The 2 no-abstract DOIs — check if OpenAlex has an `abstract_inverted_index`
-     that the mapper missed, or if the abstract is genuinely absent upstream.
-- **Acceptance:** every paper has at least a bibkey + non-zero abstract embedding,
-  or the exceptions are documented with rationale.
-
-### DQ-3 — Content fidelity (chunks vs source `.txt`): VERIFIED CLEAN · **RESOLVED 2026-06-16**
-
-**Resolution (read-only; re-ran `chunk_text` + `is_quality_chunk` on all 29
-source files and compared to stored chunks):**
-- **No paper is silently gutted.** Coverage (stored chunk chars / source chars)
-  is **109–114% for every paper** — the >100% is exactly the 64/512-word overlap
-  inflation, i.e. full retention. `stored == kept` for all 29, so the live DB
-  matches what the current pipeline produces.
-- **Only 18 chunks dropped corpus-wide, all `low_alpha`** (0 short, 0 bib),
-  concentrated in `10.1007/0-387-29555-0_13` (6) and `depositonce-5415` (10).
-  Inspected: they are **garbled OCR tables** from scanned-book sources (alpha
-  0.30–0.36, e.g. `"(u: 1) 7 < u D:= ud3424 — else H®: out.v. Family 1…"`) —
-  exactly the artefacts the alpha-ratio gate targets. **No real math content lost**
-  (the handoff's feared failure mode did not occur; math prose stays > 0.40 alpha).
-- **Head/tail alignment exact** on sampled papers: chunk 0 begins at the source
-  head (title/authors/abstract captured), last chunk ends at the source tail → no
-  front/back truncation.
-- **Note:** 0 bibliography drops across the corpus → the upstream `.txt` extraction
-  had already stripped reference lists, which is why DQ-1's citation graph relies
-  on the API/GROBID sources rather than text-mined refs.
-- **Acceptance met** for all 29 papers (not just a sample). No action needed.
-- **⚠ SCOPE LIMIT — DQ-3 covers `.txt → chunks` ONLY.** It treats each source
-  `.txt` as ground truth and verifies the *ingest pipeline* preserves it. It does
-  **not** measure the layer above — **PDF/source → `.txt`** — which was done by an
-  upstream tool in `ConformalLabpp` (outside codex-py) and is unexamined. There is
-  already evidence that layer is lossy: the 18 dropped chunks were garbled OCR
-  tables *already present in the `.txt`* (i.e. the PDF→txt step garbled the
-  table/formula regions), and the bibliographies were stripped upstream. So
-  "VERIFIED CLEAN" means **the loader faithfully preserves whatever the `.txt`
-  contains, not that no information was lost from the source paper.** See DQ-6.
-
-### DQ-6 — PDF/source → `.txt` extraction fidelity: NOT MEASURED · **MED, open**
-- **Open question:** how much did the upstream PDF→`.txt` extraction (done in
-  `ConformalLabpp`, outside codex-py) drop or garble vs the original papers?
-  Math/equations, tables, multi-column layouts, and figure captions are the usual
-  casualties of PDF text extraction — and we have direct evidence of garble
-  (DQ-3's low_alpha drops were already-garbled tables in the `.txt`).
-- **Checkable:** the 36 original PDFs are present in `…/ConformalLabpp/papers/`
-  (map each paper's `source_path` `.txt` to its sibling `.pdf` by filename stem).
-- **To investigate:** for a sample (incl. the known-bad scanned book
-  `0-387-29555-0_13` and a born-digital arXiv PDF), extract PDF text independently
-  and compare length / page-coverage to the `.txt`; spot-check whether specific
-  theorem statements & equations survive. Flag papers where the `.txt` is missing
-  large spans.
-- **Acceptance:** a per-sample source→txt coverage estimate, or a list of papers
-  whose `.txt` is materially degraded (candidates for re-extraction — note roadmap
-  **R-A/R-C** would re-ingest from PDF/`.tex` directly and largely sidestep this).
-
-**Original open question (for context):**
-- **Open question:** does the stored chunk set faithfully reconstruct each source
-  file, or did the chunker / `filter_chunks` silently drop material (e.g. an
-  abstract, a section, math-heavy passages)? The whole corpus was ingested from
-  `.txt` (`PAPERS_DIR=/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt`), so
-  the `.tex`/`.pdf` paths never ran.
-- **To investigate next:** for a sample of ~5 papers, compare
-  `len("".join(stored chunks))` against `len(source.txt)` (coverage %), and
-  eyeball the first/last chunk vs the file head/tail. Flag papers where coverage
-  is low (content lost) — F-16 dropping >X % of a paper is a signal.
-- **Acceptance:** sampled papers retain ≳ the expected fraction of source text;
-  no paper is silently gutted by the quality gate.
-
-### DQ-4 — Retrieval quality · **MEASURED 2026-06-16 — good, after fixing a P0 crash**
-
-**P0 bug found + fixed: `codex search paper` was crash-broken on the live DB.**
-The CLI passed the query embedding uncast, so pgvector raised
-`operator does not exist: vector <-> double precision[]` on every paper search.
-The working call sites (`mcp_server.py`, `wiki.py`) cast `%(emb)s::vector`; the
-CLI did not. Fixed both `<->` occurrences in `codex/cli.py search_paper`; added a
-regression guard in `tests/cli/test_cli.py` asserting the cast (mocked-DB unit
-tests can't catch the type error — this is why it survived the code audit).
-
-**Relevance verdict (post-fix): strong.** Six domain queries, both surfaces
-(paper-level abstract search + chunk-level). **rank-1 was the exactly-correct
-paper in 6/6 queries** for chunk search and 5/6 for paper-level (Laplace–Beltrami
-put the exact paper at rank-2 behind the closely-related "Polygon Laplacian Made
-Simple" — acceptable). Examples: "combinatorial Yamabe flow" → `math/0306167`
-(exact, all 5 top chunks that paper); "variational principle for Delaunay
-triangulations" → `math/0603097` (exact); "circle packing rigidity" →
-`2601.22903` (exact). Three of the DQ-1-rescued papers (`2305.10988`,
-`1005.2698`, `depositonce-5415`) now surface as top hits. The KB is trustworthy
-for lookups.
-
-**Secondary finding → reinforces DQ-2: zero-vector abstract pollution.** Papers
-with no abstract get a zero `abstract_emb`, which sits at constant L2 distance
-≈1.000 from every query and appears as filler in the paper-level tail whenever
-< 5 strongly-relevant papers exist. `1911.00966` (fully degraded, DQ-2) showed up
-at distance 1.000 in 4/6 queries. Harmless at rank>1 with a visible 1.000 score,
-but noise — and a concrete reason to fix DQ-2. Chunk-level search is immune (it
-filters `bibkey IS NOT NULL`, and 1911.00966 has no bibkey/chunks).
-
-**Note:** the MCP `search` docstring claims "hybrid dense + FTS" but the SQL is
-dense-only; `score = 1.0 - dist` can go negative (bge-m3 L2 ranges 0–2). Ranking
-is fine (monotonic); only the absolute score label is misleading.
-
-**Original open question (for context):**
-- **Open question:** end-to-end, do real queries return *relevant* results? Clean
-  chunks + a working index don't guarantee useful retrieval.
-- **To investigate next:** run a handful of domain queries through the actual
-  search path (`codex search paper "<q>"` and the chunk-level MCP `search`), e.g.
-  "discrete conformal map", "circle packing rigidity", "combinatorial Yamabe
-  flow", "discrete Laplace-Beltrami operator", and judge whether the top-5 hits
-  are on-topic and point at the right papers. Cross-check a couple against
-  `--cite-boost` to see if the boost helps or hurts on this corpus.
-- **Acceptance:** a short relevance table (query → top-5 → on/off-topic) good
-  enough to trust the KB for lookups, or a list of failure modes to fix.
-
-### DQ-5 — `ingest_paper` is NOT idempotent for DOI papers · **RESOLVED 2026-06-17** (found 2026-06-16)
-
-**Resolution (what was done):**
-- **Root-cause fix, minimal + contained:** added `openalex._normalize_doi()` and
-  applied it in `_map_paper` so the DOI branch yields the canonical **bare,
-  lower-cased** form (strips `https://doi.org/` / `http://doi.org/` / `doi:`;
-  lower-cases — DOIs are case-insensitive, matching `ingest._norm_cited_id`). The
-  no-DOI fallback (OpenAlex W-id URL) is left untouched. `_map_paper`→`fetch_paper`
-  →`ingest_paper` is the only consumer chain, so blast radius is one function.
-- **Checked nothing depended on the URL form.** It was the opposite: the full-URL
-  id silently *defeated* `_s2_id_for(paper.id)` and `_recover_abstract`'s
-  `paper.id.startswith("10.")` (a `https://…` string is neither `10.`- nor
-  `doi:`-prefixed), so S2/Crossref recovery never fired for DOI papers. The fix
-  repairs those paths too.
-- **Tests (offline, mocked DB):** closed audit **T-3** — the OpenAlex fixture now
-  emits the real URL-form DOI and asserts `paper.id` is the bare form; added
-  `_normalize_doi` unit tests (https/http/`doi:`/upper-case/already-bare) and a
-  W-id-fallback test; added ingest regression `test_ingest_doi_paper_upserts_bare
-  _canonical_id` (runs the real `_map_paper` through ingest, asserts the papers
-  upsert carries the **bare** id into `%(id)s` and uses `ON CONFLICT (id)`).
-  Verified it has teeth (fails on the pre-fix code). Full suite **348 passed**;
-  ruff + mypy --strict clean.
-- **Live idempotency confirmed (tunnel up):** re-ingested
-  `10.1007/s00454-019-00132-8` against the Jetson DB → no `UniqueViolation`;
-  `IngestResult(paper_id='10.1007/s00454-019-00132-8', citations_upserted=45)`.
-  After-state: **UPDATE in place** — `id` and `openalex_id` unchanged, **`added_at`
-  unchanged** (proves no fresh INSERT), exactly **1** row for the DOI (no URL-form
-  duplicate), citations 45→45, total papers 29→29.
-- **Unblocks R-A (GROBID re-ingest) and R-C (.tex re-ingest)** — both re-ingest
-  existing papers. **Files touched:** `codex/sources/openalex.py`,
-  `tests/sources/test_openalex.py`, `tests/ingest/test_ingest.py`.
-- **Note (out of scope, pre-existing):** arXiv papers stored as bare ids
-  (`2305.10988`) are still not round-trip idempotent if OpenAlex returns an
-  arXiv-DOI/W-id instead of the bare arXiv id — a separate canonicalization
-  concern, not this UniqueViolation. The DOI path (this finding) is fixed.
-
-**Original finding (for context):**
-- **Symptom:** re-ingesting an existing DOI paper crashes with
-  `UniqueViolation: papers_openalex_id_key`. Surfaced trying to re-ingest
-  `10.1007/s00454-019-00132-8` during the DQ-2 backfill.
-- **Root cause:** `openalex._map_paper` sets `Paper.id = data["doi"]`, which is the
-  **full URL** `https://doi.org/10.1007/…`. The stored canonical id is the **bare**
-  `10.1007/…` (post the M-1 canonical-id migration). So `INSERT … ON CONFLICT (id)`
-  finds no id match, attempts a fresh INSERT, and trips the *separate* unique
-  constraint on `openalex_id` (already held by the canonical-id row).
-- **Confirmed:** `openalex.fetch_paper("10.1007/s00454-019-00132-8").id ==
-  "https://doi.org/10.1007/s00454-019-00132-8"` (≠ stored bare id).
-- **Impact:** any re-ingest of a DOI paper aborts. This **blocks roadmap R-A
-  (GROBID on PDFs) and R-C (.tex ingest)** — both re-ingest existing papers. Also
-  implies *new* DOI ingests store full-URL ids, inconsistent with the bare-id
-  corpus (latent split-identity risk).
-- **Proposed fix (handle with care — M-1 incident territory):** normalize the DOI
-  in `_map_paper` to the canonical bare, lower-cased form (strip
-  `https://doi.org/`), matching what the M-1 migration produced. Add a re-ingest
-  idempotency test over a DOI paper. Did **not** fix inline this session — the
-  canonical-id derivation is exactly where the M-1 migration incident occurred, so
-  it needs its own focused change + a live re-ingest check.
-- **Workaround used for DQ-2:** targeted `UPDATE … SET abstract, abstract_emb`
-  instead of full re-ingest.
-
----
-
-## Roadmap — data-acquisition levers (post-audit, all free sources)
-
-These are *forward-looking acquisition improvements*, distinct from the DQ-1..DQ-4
-audit findings (which assessed existing data). They came out of a strategy
-discussion on 2026-06-16: for this Math/CS corpus the valuable data (references,
-abstracts, full text) is almost entirely **open** (arXiv, OpenAlex, Crossref, S2),
-so the quality lever is *combining more free sources*, not buying paywall access.
-Paywall/uni-login was explicitly considered and **deferred** (see R-E). Each item
-is self-contained so a cold session can pick it up.
-
-### R-A — Run GROBID reference extraction on PDFs · **CORE DONE 2026-06-17** (full sweep optional)
-
-**Resolution (what was done) — citing coverage 27/29 → 29/29 (100%); edges 920 → 1022:**
-- **References-only backfill, not full re-ingest.** Added
-  [scripts/ra_grobid_backfill.py](../../scripts/ra_grobid_backfill.py): per paper,
-  run `grobid.extract_references(pdf)`, normalize cited-ids to the canonical bare
-  form (DOIs de-URL'd + lower-cased; `arXiv:` stripped), INSERT into `citations`
-  (`ON CONFLICT DO NOTHING`). Deliberately avoids `ingest_paper(source_path=pdf)`,
-  whose PDF branch re-runs Nougat + `DELETE FROM chunks` and would overwrite the
-  DQ-3-verified-clean `.txt` chunks (and needs Nougat). Idempotent; dry-run by
-  default; 7 tests; ruff/mypy clean.
-- **GROBID provisioning:** the pinned `grobid/grobid:0.8.2` is amd64-only, but the
-  Jetson and the dev Mac are both arm64 (this is why GROBID never ran). Ran the
-  lighter CRF image (`grobid/grobid:0.8.2-crf`) on the Mac under Podman+Rosetta
-  (PDFs are local; citations written over the SSH DB tunnel). Made
-  `extract_references` timeout configurable (theses need >60s emulated). A separate
-  task is provisioning a native arm64 GROBID on the Jetson (`docs/infra/grobid-jetson.md`).
-- **Both `depositonce` theses closed:** `…depositonce-20357` (lutz-2024) **+96**
-  edges (5 in-corpus); `…depositonce-5415` (sechelmann-2016) **+6** edges — its
-  147 MB/173-page scan blew pdfalto's 120s limit, so the 9 bibliography pages were
-  extracted with PyMuPDF first (77 refs, only 6 with a DOI/arXiv). **No paper has
-  zero out-edges now.**
-- **Optional remaining (secondary acceptance):** sweep the other 27 already-covered
-  papers for *extra* GROBID refs via `python scripts/ra_grobid_backfill.py --write`
-  (the 2 book PDFs need the same bib-page extraction as sechelmann).
-
-**Original plan (for context):**
-- **✓ Unblocked (DQ-5 RESOLVED 2026-06-17):** DOI re-ingest is now idempotent, so
-  the GROBID re-ingest no longer aborts on `papers_openalex_id_key`.
-- **What:** the whole corpus was ingested from `.txt` (`PAPERS_DIR=…/papers/txt`),
-  so the GROBID PDF path never ran. Re-ingest with a PDF `source_path` per paper
-  so `codex.parsing.grobid.extract_references` parses each bibliography.
-- **Why:** extracts references the APIs lack — a *complement* to DQ-1's S2
-  supplement. Crucially, the **2 `depositonce` theses** that stayed at zero in
-  DQ-1 (no API references anywhere) almost certainly have a references section in
-  their open-access PDFs → GROBID could finally give them out-edges. Closes the
-  last 2/29.
-- **How:** no new code — ingest already merges + dedups `api_citations +
-  pdf_citations` ([codex/ingest.py:288](../../codex/ingest.py)). GROBID server is
-  already configured (`GROBID_URL=http://jetson.local:8070` in
-  `.env.jetson-ingest`). Download free arXiv PDFs, re-ingest with `source_path`.
-- **Acceptance:** the 2 theses gain references; published papers gain
-  GROBID-parsed refs beyond OpenAlex/S2 coverage; citing coverage 27/29 → 29/29.
-
-### R-B — Add Crossref as a third citation + abstract source · **DONE 2026-06-17**
-
-**Resolution (what was done):**
-- **`crossref.fetch_references(doi)`** added (mirrors `semanticscholar.fetch_references`):
-  GETs `/works/{doi}`, returns one `Citation` per `message.reference[]` entry that
-  carries a registered `DOI` (book/`unstructured`-only refs are skipped — they
-  can't join the graph). `citing_id` is the queried DOI; the ingest supplement
-  rewrites it to `paper.id`.
-- **Wired as the third leg of the citation fallback chain** in `ingest.py`:
-  OpenAlex-empty → S2 → **Crossref** (`_crossref_reference_supplement`, DOI papers
-  only — Crossref's works endpoint is DOI-keyed). cited DOIs pass through
-  `_norm_cited_id` (bare, lower-cased). Network/parse failures degrade to [].
-- **Tests:** `fetch_references` unit tests (DOI edges / `unstructured` skipped / no
-  `reference` key / 404) + ingest tests asserting the Crossref leg fires when
-  OpenAlex+S2 are empty (cited DOI normalised) **and** does *not* fire when OpenAlex
-  has citations. Suite **361 green**, ruff/mypy clean. The DQ-2 abstract half
-  (`fetch_abstract`) already shipped earlier.
-- **Live-validated:** `fetch_references` against the real API returned 33 / 40 / 24
-  DOI-bearing refs for `10.1007/s00454-019-00132-8` / `10.1145/3132705` /
-  `10.1111/cgf.13931` (Springer / ACM / Wiley).
-- **No live corpus change (by design):** as a *fallback*, Crossref fires only when
-  OpenAlex **and** S2 are both empty. Post-DQ-1/R-A the only such papers were the 2
-  `depositonce` theses, which Crossref doesn't index either — so R-B is a
-  forward-looking third leg for future DOI ingests, not a backfill. (Harvesting
-  Crossref's *unique* refs for already-covered papers would need an always-merged
-  supplement instead — a deliberate scope choice not taken, per the doc's "fallback
-  chain" spec.) **Files:** `codex/sources/crossref.py`, `codex/ingest.py`, + tests.
-
-**Original plan (for context):**
-- **What:** new `codex.sources.crossref` client hitting Crossref REST
-  `/works/{doi}` for the `reference` array and `abstract`.
-- **Why:** Crossref carries **publisher-deposited references** for many DOIs (free,
-  polite pool via `mailto`) — a third leg after OpenAlex/S2. It also recovers
-  abstracts OpenAlex is contractually barred from redistributing (likely the 2
-  Springer DOIs missing abstracts in DQ-2). cited_ids are DOIs → flow through the
-  existing graph resolver unchanged.
-- **How:** mirror `codex/sources/openalex.py` (tenacity retry, polite pool). Extend
-  the ingest fallback chain to OpenAlex-empty → S2 → Crossref (same
-  `_s2_reference_supplement` shape). Add unit tests with mocked HTTP.
-- **Acceptance:** Crossref recovers refs and/or an abstract for ≥1 paper where
-  OpenAlex+S2 are empty.
-
-### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **DONE 2026-06-17**
-
-**Resolution (what was done) — all 13 arXiv papers re-ingested from `.tex`:**
-- **Multi-file flatten:** `arxiv.fetch_source` now inlines `\input`/`\include`
-  (`tex.flatten_inputs`) so multi-file projects yield the full body, not the
-  primary file's include skeleton (e.g. math/0603097 = 15 files / 12 includes;
-  2305.10988 = 6 / 5). Also handles legacy single-file **bare-gzipped** `.tex`
-  (no tar wrapper) for old math/* papers (commit 121b582).
-- **Section-aware chunking:** new `tex.chunk_sections` returns `(title, chunk)`
-  pairs (chunks never span a `\section`); the ingest `.tex` path labels the stored
-  `section` from the real heading instead of running `classify_section` on a
-  header-less word-window. Quality-gated as pairs so labels stay aligned.
-- **Unblocked by the arXiv half of DQ-5** (commit f504ca6): `_canonical_id`
-  strips the `10.48550/arxiv.` DOI prefix so an arXiv paper's id is the bare
-  arXiv id — without it, re-ingest tripped `papers_openalex_id_key` (confirmed
-  live, then fixed; idempotent re-ingest verified).
-- **Outcome:** `section` column gained signal — ~34/329 arXiv chunks now
-  intro/theorem/proof (was ~all `body`); math/0603097 `{body:31}`→`{body:29,
-  proof:9}`, 2310.17529 gained proof+intro, etc. Math papers with descriptively
-  titled sections stay `body` (classify_section's fixed vocab) — modest but real
-  (R-12). No paper gutted (chunk counts stable/▲); F-16 gate unchanged.
-- **Files:** `codex/parsing/tex.py`, `codex/sources/arxiv.py`, `codex/ingest.py`,
-  `codex/sources/openalex.py`, `scripts/rc_tex_reingest.py` (driver, dry-run
-  default), + tests. Full suite **370 passed**; ruff/mypy clean. Branch
-  `feat/tex-ingest` (commits 1da81c7, 121b582, f504ca6).
-
-**Original plan (for context):**
-- **What:** re-ingest from arXiv LaTeX source through the existing
-  `codex.parsing.tex.latex_to_text` path instead of flattened `.txt`.
-- **Why:** highest-fidelity input — preserves math, structure, and section
-  boundaries. Directly serves **DQ-3** (content fidelity) and audit **R-12** (the
-  `section` column is low-signal because word-window `.txt` chunks rarely start at
-  a real header; `.tex` headers fix this).
-- **How:** download free arXiv source tarballs, feed `.tex` to ingest with
-  `source_path`. Handle multi-file projects (main `.tex` + includes).
-- **Acceptance:** `section` column gains signal (fewer `body`-only); DQ-3 coverage
-  vs source improves; no regression in the F-16 quality gate.
-
-### R-D — F-15: warn on low *citing-paper* coverage, not just paper count · **DONE 2026-06-17**
-
-**Resolution (what was done):**
-- Added `graph.citing_coverage(graph, known_ids)` → `(papers_with_out_edges,
-  total)`, a `graph_min_citing_coverage` setting (default 0.8), and wired both into
-  `codex graph report`: it now prints `Citing coverage: N/M papers with out-edges
-  (P%)` (and in `--json`), and emits a `Warning` when the share is below the
-  threshold ("low citing coverage weakens PageRank/coupling"). Separate from the
-  existing `graph_min_corpus_size` (total-count) warning.
-- Tests: `citing_coverage` unit tests + CLI tests (line shown; warning fires at
-  low coverage; suppressed at 100%; JSON field). Full suite green; ruff/mypy clean.
-- **Live:** `codex graph report` on the corpus prints `Citing coverage: 29/29
-  papers with out-edges (100%)` with no warning (post-R-A). Branch
-  `feat/graph-coverage-warning`. Files: `codex/config.py`, `codex/graph.py`,
-  `codex/cli.py`, + tests.
-
-**Original plan (for context):**
-- **What:** the open DQ-1 sub-item: `graph_min_corpus_size` only flags total paper
-  count. Add a warning when the share of papers with ≥1 out-edge is low (e.g.
-  < 80%), since that is what actually starves PageRank/coupling.
-- **Why:** at 17/29 the graph silently ran on 59% of the corpus with no signal.
-  Post-DQ-1 it's 27/29 (93%) — but the guard should catch future regressions.
-- **Acceptance:** `codex graph report` surfaces citing-coverage % and warns below
-  threshold.
-
-### R-E — Paywall / institutional access · **DEFERRED — only for a paywall-only expansion**
-- **Decision (2026-06-16):** a uni login does **not** help the current corpus —
-  every DQ-1/DQ-2 gap is in openly-available works (arXiv preprints, OA theses),
-  and all metadata/citation sources are free APIs. Revisit *only* if the corpus
-  expands to papers that exist solely behind a paywall with no preprint; then
-  institutional access → full-text PDF → GROBID refs (R-A) + clean chunks (R-C).
-- **ToS caveat:** manual download of individual papers you have legitimate access
-  to is fine; **automated bulk download through a uni proxy (EZproxy/Shibboleth)
-  violates most publisher terms and can get the whole institution's access
-  revoked.** Do not wire a publisher login into the ingest pipeline.
-
-### R-F — Richer section labels: store real `\section` titles · **DONE 2026-06-17**
-
-**Resolution (what was done):**
-- **Why:** R-C labelled `.tex` chunks by mapping the real `\section` heading through
-  `classify_section`'s fixed vocab, so descriptively-titled Math sections collapsed
-  to `body` (only **34/329 = 10 %** non-`body`). A research pass confirmed the
-  `section` column is **write-only** (nothing filters on the vocab — `mcp_server.py`
-  and `wiki.py` don't even SELECT it), so storing free-text titles is safe.
-- **Change:** new `quality.section_label(title, content)` (+ `_clean_title`) — keep the
-  controlled bucket (intro/theorem/proof/abstract/bibliography) when the heading maps
-  to one, else store the cleaned real title ("the flip algorithm", "main results", …).
-  The `.tex` ingest path uses it; `.pdf`/`.txt`/`run_quality_pass` keep
-  `classify_section` unchanged. `_clean_title` strips LaTeX commands/accents/ties +
-  leading numbering, lower-cases, truncates at a word boundary. ~1 helper + 1 ingest
-  line; no schema change (column already free `TEXT`).
-- **Result (live):** re-applied to the 13 arXiv papers → non-`body` **34/329 (10 %) →
-  314/329 (95 %)**. Only `math/0306167` (old AMS-TeX, no `\section{}`) stays `body`.
-- **Tests:** `section_label` / `_clean_title` units + `.tex` ingest stores a
-  descriptive heading verbatim. Full suite 377 passed; ruff/mypy clean. Branch
-  `feat/section-labels` (off `feat/tex-ingest`).
-- **Accent re-clean — DONE 2026-06-17.** The `_clean_title` polish (`m\"obius`→`mobius`,
-  `~` ties, word-boundary truncation) landed in code, and the existing live labels were
-  re-cleaned in place (apply `_clean_title` to any `chunks.section` containing `\` or
-  `~`; **8 labels** fixed, no re-embed). Verified: **0** labels with LaTeX artifacts
-  remain; non-`body` holds at **314/329 (95 %)**. R-F fully complete.
-
----
-
-## Priority for the next session
-1. ~~**DQ-1**~~ — **DONE 2026-06-16** (live DB at 920 citations, 27/29 citing
-   coverage; remedy wired into ingest).
-2. ~~**DQ-4**~~ — **DONE 2026-06-16** (P0 `search paper` crash fixed; relevance
-   verified strong; surfaced the zero-vector pollution that motivates DQ-2).
-3. ~~**DQ-2**~~ — **DONE 2026-06-16** (1911.00966 fully recovered; s00454 abstract
-   from S2; book chapter documented; recovery wired into ingest).
-4. ~~**DQ-3**~~ — **DONE 2026-06-16** (content fidelity clean; full coverage, only
-   18 OCR-junk chunks dropped, no math lost). **Assessment axis DQ-1..DQ-4 complete.**
-5. ~~**DQ-5**~~ — **DONE 2026-06-17** (DOI id normalized to bare form in
-   `openalex._map_paper`; live re-ingest of `10.1007/s00454-019-00132-8` idempotent
-   — UPDATE in place, `added_at` unchanged; suite 348 green). **Unblocks R-A/R-C.**
-6. **Roadmap levers** (see section above — now all unblocked): **R-A** (GROBID on
-   arXiv PDFs — closes the last 2 zero-citation theses), **R-B** (Crossref refs —
-   abstract half already shipped in DQ-2), **R-C** (`.tex` ingest — serves DQ-3),
-   **R-D** (F-15 coverage warning, quick). **R-E** (paywall) deferred.
-
-DQ-1..DQ-4 are *data*/assessment work; DQ-5 + roadmap R-A..R-D add **new code**
-(canonical-id fix, re-ingest scripts, Crossref references, a `.tex` path, a graph
-warning) on top of the already-remediated pipeline (see the AUDIT-* docs and PRs
-#12–#14). The DQ-1/DQ-2 ingest wiring + Crossref/arXiv abstract recovery already
-landed this session.
-
----
-
-## Appendix — reproduce the baseline scan
-
-```python
-import os, psycopg, statistics
-from collections import Counter, defaultdict
-from psycopg.rows import dict_row
-from codex.config import Settings
-from codex.quality import is_quality_chunk, _bib_score
-
-s = Settings()
-with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row) as c:
-    papers = c.execute("SELECT id, bibkey, title, abstract, year, authors, openalex_id FROM papers").fetchall()
-    chunks = c.execute("SELECT paper_id, content, section FROM chunks").fetchall()
-    cit = c.execute("SELECT citing_id, count(*) n FROM citations GROUP BY citing_id").fetchall()
-
-# zero-citation papers (DQ-1)
-cited = {r["citing_id"] for r in cit}
-zero = [p["id"] for p in papers if p["id"] not in cited]
-# metadata gaps (DQ-2): p["abstract"] empty / p["bibkey"] None / not p["authors"]
-# F-16 re-gate (baseline): [ch for ch in chunks if not is_quality_chunk(ch["content"], settings=s)]
-# section dist (R-12): Counter(ch["section"] or "(null)" for ch in chunks)
-```

docs/coldstart-evaluate-review-release.md

@@ -1,59 +0,0 @@
-# Cold-start session prompt — evaluate open points · review · release
-
-Paste the block below into a fresh Claude Code session (it assumes no memory of prior work).
-
----
-
-You are picking up **codex-py** cold. The data-quality roadmap (**DQ-1–DQ-5 + R-A–R-F**)
-was merged to `main` via **PR #16** (merge commit `5aa2f7d`). Your job this session, in
-order: **(1) evaluate the remaining open points, (2) code-review the released state,
-(3) cut a release.** Read the two handoff docs first, then work the tasks. Ask before any
-destructive action (branch deletion) or before publishing the release.
-
-## Orient yourself first
-- Repo: `/Users/tarikmoussa/Desktop/codex-py`, branch `main` (at the PR #16 merge).
-- Read: `docs/SESSION-2026-06-17-open-items.md` (open items + branch-cleanup lists),
-  `docs/audit/DATA-QUALITY-2026-06-15.md` (DQ-1–DQ-5 + R-A–R-F resolution blocks),
-  `docs/infra/jetson-ssh-stability.md` (infra TODO), `docs/infra/grobid-jetson.md`.
-- Live DB: Jetson Postgres via SSH tunnel
-  `ssh -o ServerAliveInterval=15 -f -N -L 5433:localhost:5432 alfred@jetson.local`;
-  creds in `.env.jetson-ingest`. **The tunnel is chronically flaky** — prefer short,
-  idempotent ops; for long live runs use tmux **on the Jetson** (no tunnel needed there).
-- Tests: `PYTHONPATH=. .venv/bin/python -m pytest -q`. **Important:** importing `codex`
-  eagerly loads the embedding libs (~2 min) so the suite is slow, and this sandbox
-  auto-backgrounds + reaps long commands. **Launch ONE pytest run and issue no other
-  commands until it notifies you** — that lets it finish (polling its output file with a
-  read-only Read is fine; a new Bash command reaps it). Baseline after PR #16: 408 green.
-- Remote: `origin` = `git.eulernest.eu/user2595/codex-py`. Use the **gitea MCP** for
-  PRs/tags/releases (tools `mcp__gitea-eulernest__*`).
-
-## Task 1 — Evaluate open points
-- **Branch cleanup:** confirm these are fully merged into `main`, then propose deletion
-  (local + remote): `feat/tex-ingest`, `feat/crossref-references`,
-  `feat/graph-coverage-warning`, `feat/section-labels`, `chore/infra-ssh-stability`,
-  `audit/loop-1`, `integration/roadmap`; plus pre-session stale: `feat/F-15-literature-graph`,
-  `feat/F-16-chunk-quality`, `fix/audit-wave-1/2/3`, `scratch/spike-embed`.
-- **Deferred review findings** (intentionally left in PR #16 — see the `fix(review)`
-  commit): #3 (`chunk_sections` drops pre-`\section` text), #8 (script id-normalizer
-  duplication), #9 (arXiv cited-id case), #11 (gzip heuristic), #12 (title truncation).
-  Re-assess whether any is worth doing now.
-- **Infra TODO:** tmux-on-Jetson for stable long ops — documented, not implemented.
-- **DQ-6** (upstream PDF→`.txt` fidelity) — flagged, believed mooted by R-A/R-C; confirm.
-- If the tunnel is up, a quick corpus scan for any new data-quality gap.
-
-## Task 2 — Review
-- Run `/code-review` on `main` (the released state) and confirm no regressions.
-- Confirm the full suite is green (`pytest -q`, honoring the launch-and-wait rule), plus
-  `ruff check`, `ruff format --check`, `mypy codex/ scripts/`.
-
-## Task 3 — Release
-- Only after the review + suite are green.
-- Bump the version in `pyproject.toml` (currently `0.1.0`; the roadmap is a substantial
-  feature set → `0.2.0`). Commit on a branch (do not commit straight to `main`) and PR it,
-  or tag the merge commit directly if that matches the team's flow.
-- Write release notes summarizing DQ-1–DQ-5 + R-A–R-F + the audit-remediation reconcile;
-  corpus impact: **citations 590→1075, citing coverage 29/29, section signal 10%→95%**.
-- Create the tag + Gitea release via the gitea MCP; confirm it's published.
-
-Deliverable: a short report of (1) the open-point decisions, (2) the review outcome,
-(3) the published release tag/URL.

docs/infra/grobid-jetson.md

@@ -1,192 +0,0 @@
-# GROBID on the Jetson (aarch64) — deploy runbook
-
-**Status:** ACTIVE
-**Host:** `alfred@jetson.local` — NVIDIA Jetson Orin Nano (aarch64, 8 GB RAM, 8 GB swap, 6 cores)
-**Service:** GROBID reference extraction for roadmap **R-A** (`scripts/ra_grobid_backfill.py`)
-**Last updated:** 2026-06-17
-**Verified:** 2026-06-17 — `grobid/grobid:0.9.0-crf` pulled as native `arm64`, `/api/isalive`→`true`,
-end-to-end `extract_references` on `lutz-2024-thesis.pdf` → 116 refs (101 with DOI/arXiv) in ~36s.
-
----
-
-## Why this image (`grobid/grobid:0.9.0-crf`)
-
-GROBID had never come up on the Jetson because `infra/docker-compose.yml` pinned
-`grobid/grobid:0.8.2`, and **every `0.8.x` tag (and the full `0.9.0`) is published
-amd64-only**. A Docker tag resolves to a multi-arch *manifest list*; on aarch64
-the `0.8.2` list has no `linux/arm64` entry, so there is simply nothing to pull or
-run (hence `curl localhost:8070` → connection refused, nothing listening).
-
-The fix is to select a tag whose manifest list actually contains a `linux/arm64`
-build:
-
-| Candidate                     | arm64? | Size    | Notes                                  |
-|-------------------------------|:------:|--------:|----------------------------------------|
-| `grobid/grobid:0.8.2`         | ✗      | ~9.5 GB | full DL image, amd64-only (old pin)    |
-| `grobid/grobid:0.9.0` / `-full`| ✗     | ~10 GB  | full DL image, amd64-only              |
-| **`grobid/grobid:0.9.0-crf`** | **✓**  | ~500 MB | **CRF-only, official, multi-arch** ←   |
-| `lfoppiano/grobid:0.9.0-crf`  | ✓      | ~500 MB | same build, community namespace (alt.) |
-
-We use **`grobid/grobid:0.9.0-crf`**:
-
-- **CRF-only** is sufficient for reference extraction (`/api/processReferences`)
-  and is ~500 MB vs ~10 GB for the deep-learning image.
-- **Native arm64** — no qemu emulation (too slow / RAM-heavy on a Jetson).
-- **Official namespace**, **pinned version** (not `latest-crf`, which is a moving
-  target). arm64 builds only exist from `0.9.0-crf` onward — there is no arm64
-  `0.8.x`, so this is a forced (but compatible) bump from the old `0.8.2` pin. The
-  TEI that `codex/parsing/grobid.py` parses (`biblStruct`, `persName`,
-  `idno[@type='DOI']`, …) is unchanged across the bump.
-
-> Caveat from upstream: the arm64 image is documented as "tested only on macOS,
-> not linux/arm64". The container is the *same* `linux/arm64` ELF either way, so
-> bare-metal Jetson works; the end-to-end check below is what de-risks it.
-
----
-
-## One-time prerequisites
-
-### 1. Docker permission for `alfred`
-
-The SSH user `alfred` is in the `sudo` group but **not** the `docker` group, so
-`docker …` fails with `permission denied … /var/run/docker.sock`. Grant access
-once (requires the sudo password):
-
-```bash
-ssh alfred@jetson.local
-sudo usermod -aG docker alfred      # add to docker group (persistent)
-```
-
-Then **start a fresh login** so the new group takes effect (group membership is
-only re-evaluated at login — an existing session/`ssh` won't see it; a *new*
-`ssh` connection will):
-
-```bash
-exit && ssh alfred@jetson.local
-docker ps                            # should now work without sudo
-```
-
-(Alternatively run all `docker` commands below under `sudo`, but group membership
-is cleaner and lets `ra_grobid_backfill.py` / curl checks be driven over plain SSH.)
-
-### 2. Docker Compose v2 plugin (arm64)
-
-This host has Docker Engine but **not** the Compose v2 CLI plugin (on Linux they
-are separate packages; only Docker Desktop bundles Compose). Without it
-`docker compose …` fails with `docker: unknown command: docker compose`. Install
-the arm64 plugin into the user-local plugin dir (no sudo needed):
-
-```bash
-mkdir -p ~/.docker/cli-plugins
-curl -sSL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-aarch64 \
-  -o ~/.docker/cli-plugins/docker-compose
-chmod +x ~/.docker/cli-plugins/docker-compose
-docker compose version               # confirm it resolves
-```
-
----
-
-## Deploy
-
-GROBID is **service-scoped** on purpose: Postgres (`papers-db`) is already running
-on this host, so we bring up only the `grobid` service and never `docker compose up`
-the whole stack (that would try to start a second `papers-db`).
-
-```bash
-# copy this repo's compose file to the Jetson (first time only)
-scp infra/docker-compose.yml infra/schema.sql alfred@jetson.local:~/codex-infra/
-
-ssh alfred@jetson.local
-cd ~/codex-infra
-docker compose up -d grobid          # pulls grobid/grobid:0.9.0-crf (arm64), starts papers-grobid
-docker compose logs -f grobid        # wait for "Started ... in N seconds" (model load ~20–40s)
-```
-
-The `deploy.resources.limits.memory: 4g` cap in the compose file is honored by
-Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves
-headroom alongside Postgres. The `restart: "no"` policy means GROBID does **not**
-auto-start on boot — see "Run on-demand" below.
-
----
-
-## Run on-demand (stop when idle)
-
-GROBID is only needed **during reference extraction** (`scripts/ra_grobid_backfill.py`);
-nothing else in the stack uses it. Idle it still holds ~3.5 GB, so on the 8 GB Jetson
-keep it stopped and start it only for an R-A run:
-
-```bash
-ssh alfred@jetson.local 'docker start papers-grobid'        # ~30s model load
-curl -s --retry 30 --retry-delay 2 --retry-connrefused \
-  http://jetson.local:8070/api/isalive                      # wait for: true
-
-# ... run R-A from the Mac ...
-PYTHONPATH=. python scripts/ra_grobid_backfill.py --write
-
-ssh alfred@jetson.local 'docker stop papers-grobid'         # reclaim ~3.5 GB
-```
-
-First-time deploy uses `docker compose up -d grobid` (above); afterwards the
-container persists, so `docker start/stop papers-grobid` is all you need.
-
-> Why on-demand rather than a lighter tool: refextract was evaluated head-to-head
-> on the Jetson and rejected — far worse DOI recall on this math corpus (lutz
-> thesis: GROBID **101** usable IDs vs refextract **2**), and it was *slower* per
-> PDF despite a smaller footprint. GROBID stays; running it on-demand removes its
-> only real downside (idle RAM).
-
----
-
-## Verify `GET /api/isalive` → `true`
-
-```bash
-# on the Jetson
-curl -s http://localhost:8070/api/isalive          # => true
-
-# from the Mac (LAN) — same value the corpus' GROBID_URL points at
-curl -s http://jetson.local:8070/api/isalive     # => true
-```
-
-Anything other than `true` (timeout / connection refused / `false`) means the
-container isn't up — check `docker compose logs grobid` and that nothing else
-holds port 8070.
-
----
-
-## End-to-end reference-extraction check + running R-A
-
-With `GROBID_URL=http://jetson.local:8070` (already set in `.env.jetson-ingest`):
-
-```bash
-# one PDF, references only — should print N>0 parsed references
-PYTHONPATH=. python - <<'PY'
-from codex.parsing.grobid import extract_references
-refs = extract_references("/Users/tarikmoussa/Desktop/ConformalLabpp/papers/lutz-2024-thesis.pdf",
-                          grobid_url="http://jetson.local:8070")
-print(f"{len(refs)} references; first DOI/arXiv:",
-      next(((r['doi'], r['arxiv_id']) for r in refs if r['doi'] or r['arxiv_id']), None))
-PY
-```
-
-Then run the references-only backfill (DB tunnel up — see
-`docs/audit/DATA-QUALITY-2026-06-15.md`):
-
-```bash
-ssh -N -L 5433:localhost:5432 alfred@jetson.local &     # DB tunnel
-cp .env.jetson-ingest .env
-PYTHONPATH=. python scripts/ra_grobid_backfill.py --zero-edge-only   # dry-run first
-PYTHONPATH=. python scripts/ra_grobid_backfill.py --write            # apply
-```
-
----
-
-## Troubleshooting
-
-- **Container killed mid-request / exit 137** — OOM. The CRF image needs ~3 GB for
-  references; raise the compose `memory` limit or check Postgres isn't starving the
-  box (`free -h`, `docker stats papers-grobid`).
-- **`isalive` true but `processReferences` 500s** — usually a `pdfalto` failure on a
-  malformed PDF; confirm with a clean PDF (`lutz-2024-thesis.pdf`) and check
-  `docker compose logs grobid`.
-- **`permission denied … docker.sock`** — the docker-group prerequisite above wasn't
-  applied, or the SSH session predates it (re-login).

docs/infra/jetson-ssh-stability.md

@@ -1,66 +0,0 @@
-# Infra TODO — stable long-running ops against the Jetson (tmux + autossh)
-
-**Status:** OPEN (infra). **Filed:** 2026-06-17. **Priority:** MED — it slows live ops
-but has a known workaround (short/idempotent ops).
-
-## Problem (observed)
-
-Live data ops run on the **Mac** and reach the Jetson Postgres through an SSH
-port-forward tunnel (`ssh -f -N -L 5433:localhost:5432 alfred@jetson.local`,
-`DATABASE_URL` → `localhost:5433`). During the R-A / R-C / R-F work this tunnel
-**dropped repeatedly mid-run**: any operation longer than a few minutes (corpus
-re-ingest, the bge-m3 embed loop, the GROBID sweep) had a high chance of failing
-partway when the tunnel died, leaving the work half-applied. At one point the Jetson
-went **fully offline** (ping 100 % loss, `ssh:22` timeout) mid-operation.
-
-`ingest_paper()` and the backfill scripts open one DB connection for the whole run, so
-a tunnel drop raises `OperationalError: connection refused` and aborts — partial state,
-manual retry. Adding `ServerAliveInterval` to the tunnel helped only marginally.
-
-## Recommended fix
-
-**1. Run long ops *on the Jetson* inside `tmux` (primary fix).**
-The Jetson is where Postgres (and GROBID) live, so running ingest there means the DB is
-`localhost` — **no tunnel at all** — and `tmux` keeps the process alive across SSH
-disconnects (detach / reattach to monitor). This removes the tunnel from the critical
-path for every long-running write.
-
-- [ ] Confirm the `codex` env is deployable on the Jetson (Python 3.12+, deps incl. the
-      bge-m3 embedder; the Orin Nano GPU should handle it — GROBID already runs there).
-      If not, stand up a venv / container for it.
-- [ ] Standard pattern: `ssh alfred@jetson`, then
-      `tmux new -s codex` → `set -a; source .env; set +a` →
-      `PYTHONPATH=. python scripts/<job>.py --write` → detach (`Ctrl-b d`).
-      Reattach later with `tmux attach -t codex`. The job survives the SSH drop.
-- [ ] Point the Jetson-side `DATABASE_URL` at `localhost:5432` directly (no `:5433`).
-- [ ] Document this as the default for any multi-minute write (re-ingest, sweeps,
-      embed loops) in `docs/audit/DATA-QUALITY-2026-06-15.md` "How to reach the data".
-
-**2. `autossh` for when a tunnel *is* needed (secondary).**
-For interactive/dev use from the Mac (psql, quick probes, read-only queries), replace
-the plain `ssh -f -N -L` with `autossh -M 0 -f -N -o ServerAliveInterval=15 -o
-ServerAliveCountMax=3 -L 5433:localhost:5432 alfred@jetson` so a dropped tunnel
-auto-reconnects. NB: autossh only restarts the *tunnel* — an in-flight transaction
-still fails, so this is for short/interactive use, not long writes (use tmux-on-Jetson
-for those).
-
-**3. (Optional, code) make backfill scripts resilient.**
-Lower-value once (1)/(2) are in place, but: open a fresh `get_conn()` per paper (not one
-for the whole run) and/or wrap the per-item write in a small retry, so a transient drop
-costs one item, not the whole batch. The R-A backfill already does per-paper connections;
-`rc_tex_reingest.py` opens a connection per paper too — but `ingest_paper` itself holds
-one connection for its whole body, which is the unit that fails.
-
-## Why this matters
-
-The chronic flakiness directly cost time this session and left one cosmetic task
-unfinished (the R-F accent re-clean — see DATA-QUALITY-2026-06-15.md R-F "Pending"). The
-**robust pattern that already works** is short, idempotent operations (per-paper, or a
-single quick `UPDATE`); tmux-on-Jetson makes even the long ops safe.
-
-## Acceptance
-
-- A documented, repeatable way to run a full corpus re-ingest that survives an SSH drop
-  (tmux-on-Jetson), verified by detaching/reattaching across a disconnect.
-- The DATA-QUALITY doc's "How to reach the data" section recommends tmux-on-Jetson for
-  long writes and `autossh` for interactive tunnels.

infra/docker-compose.yml

@@ -28,34 +28,20 @@ services:
     restart: unless-stopped
 
   grobid:
-    # CRF-only image: sufficient for reference extraction (/api/processReferences)
+    # CRF-only image is sufficient for reference extraction and is lighter than
-    # and ~500MB vs ~10GB for the full deep-learning variant.
+    # the full deeplearning variant. Check https://hub.docker.com/r/grobid/grobid
-    #
+    # for the latest 0.8.x tag before deploying.
-    # The `-crf` tag is also the ONLY GROBID variant published as a multi-arch
+    image: grobid/grobid:0.8.2
-    # manifest that includes linux/arm64 — the full image (grobid/grobid:0.9.0)
-    # and every 0.8.x tag are amd64-only. So this is what lets GROBID run
-    # *natively* on the aarch64 Jetson (no slow/RAM-heavy qemu emulation). arm64
-    # builds start at 0.9.0-crf; there is no arm64 0.8.x.
-    # Deploy + docker-permission fix: docs/infra/grobid-jetson.md
-    image: grobid/grobid:0.9.0-crf
     container_name: papers-grobid
     ports:
       - "8070:8070"
-    init: true            # reap zombie children (GROBID docs recommend --init)
+    restart: unless-stopped
-    ulimits:
+    # GROBID is RAM-hungry; uncomment to cap usage on constrained hardware
-      core: 0             # disable core dumps (GROBID docs: --ulimit core=0)
+    # (e.g. Nvidia Jetson):
-    # On-demand only: GROBID is needed solely during reference extraction
+    # deploy:
-    # (scripts/ra_grobid_backfill.py), not for normal corpus operations — start it
+    #   resources:
-    # for an R-A run, stop it after to reclaim ~3.5GB. "no" = no auto-start on boot.
+    #     limits:
-    # See docs/infra/grobid-jetson.md ("Run on-demand").
+    #       memory: 4g
-    restart: "no"
-    # GROBID is RAM-hungry; cap usage on the constrained Jetson (Orin Nano, 8GB
-    # RAM shared with Postgres). 3GB suffices for references; 4g leaves headroom.
-    # Honored by `docker compose up` under Compose v2 (no swarm needed).
-    deploy:
-      resources:
-        limits:
-          memory: 4g
 
 volumes:
   pgdata:

infra/reingest_canonical_ids.sh

@@ -1,141 +0,0 @@
-#!/usr/bin/env bash
-# One-time migration to canonical paper IDs (audit C-7).
-#
-# Before the C-7 fix, ingest stored OpenAlex's URL-form doi/id as papers.id
-# (e.g. "https://doi.org/10.48550/arxiv.math/0603097"). After the fix, ingest
-# keys papers.id on the bare caller id ("math/0603097"). The existing rows keep
-# their old URL-form primary keys, so a plain re-ingest would DUPLICATE every
-# paper (new bare-id row alongside the old URL-id row). This script wipes the
-# corpus and rebuilds it on canonical ids.
-#
-# DESTRUCTIVE: `TRUNCATE papers CASCADE` removes papers + chunks + citations +
-# formulas + figures + paper_identifiers (code_links.paper_id is set NULL). All
-# content is rebuilt by ingest_all.sh, which re-fetches OpenAlex and re-embeds
-# (minutes for ~36 papers).
-#
-# Prerequisites: SSH tunnel on :5433 (see ingest_all.sh) and .env.jetson-ingest.
-# Usage: bash infra/reingest_canonical_ids.sh
-set -euo pipefail
-
-CODEX_DIR="$(cd "$(dirname "$0")/.." && pwd)"
-ENV_FILE="$CODEX_DIR/.env.jetson-ingest"
-PY="$CODEX_DIR/.venv/bin/python"
-TUNNEL_PORT=5433
-
-# ── 1. Preconditions ─────────────────────────────────────────────────────────
-if ! nc -z localhost "$TUNNEL_PORT" 2>/dev/null; then
-  echo "ERROR: SSH tunnel not active on :$TUNNEL_PORT"
-  echo "Run: ssh -f -N -L 5433:localhost:5432 alfred@jetson.local"
-  exit 1
-fi
-[[ -f "$ENV_FILE" ]] || { echo "ERROR: $ENV_FILE missing"; exit 1; }
-[[ -x "$PY" ]] || { echo "ERROR: venv python not found at $PY (run uv sync)"; exit 1; }
-# Load DATABASE_URL into the environment (not echoed — keeps the password out of logs).
-set -a; source "$ENV_FILE"; set +a
-
-# run_sql <SQL>: execute one statement via psycopg, print any result rows.
-# Reads DATABASE_URL from the environment; suppresses the DSN on error.
-run_sql() {
-  "$PY" - "$1" <<'PYEOF'
-import os, sys, psycopg
-from psycopg.rows import dict_row
-try:
-    with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row, connect_timeout=10) as c:
-        cur = c.execute(sys.argv[1])
-        if cur.description:
-            for row in cur.fetchall():
-                print("   " + "  ".join(f"{k}={v!r}" for k, v in row.items()))
-        c.commit()
-except Exception as e:
-    print(f"   DB ERROR: {type(e).__name__} (details suppressed to avoid DSN leak)")
-    sys.exit(1)
-PYEOF
-}
-
-# ── 2. BEFORE snapshot ───────────────────────────────────────────────────────
-echo "── BEFORE migration ──────────────────────────────────────────"
-run_sql "SELECT count(*) AS papers, count(*) FILTER (WHERE starts_with(id, 'https://')) AS url_form_ids FROM papers"
-echo "   sample ids:"
-run_sql "SELECT id FROM papers ORDER BY added_at LIMIT 3"
-
-# ── 2b. Preflight: verify the schema is re-ingestable BEFORE wiping anything ──
-# ingest writes chunks.section (F-16). On a live DB that predates it the column
-# is missing, and the app user cannot ALTER tables owned by 'postgres' (audit
-# M-1). Check FIRST so we never TRUNCATE and then fail mid-re-ingest.
-echo "── Preflight: schema readiness ───────────────────────────────"
-SECTION_OK="$("$PY" - <<'PYEOF'
-import os, psycopg
-try:
-    with psycopg.connect(os.environ["DATABASE_URL"], connect_timeout=10) as c:
-        cols = [r[0] for r in c.execute(
-            "SELECT column_name FROM information_schema.columns WHERE table_name='chunks'").fetchall()]
-        print("yes" if "section" in cols else "no")
-except Exception:
-    print("error")
-PYEOF
-)"
-if [[ "$SECTION_OK" != "yes" ]]; then
-  echo "ABORT — schema not ready: chunks.section is missing (audit M-1)."
-  echo "The app user cannot add it (core tables are owned by 'postgres'). Sync the"
-  echo "schema with a privileged role, then re-run this script. Either:"
-  echo ""
-  echo "  (a) MIGRATION_DATABASE_URL=postgresql://postgres:PASS@HOST:PORT/papers \\"
-  echo "        \"$CODEX_DIR/.venv/bin/codex\" migrate     # applies schema.sql idempotently"
-  echo "  (b) sudo -u postgres psql -d papers \\"
-  echo "        -c \"ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT;\""
-  echo ""
-  echo "Nothing was changed — no TRUNCATE was issued."
-  exit 1
-fi
-echo "   chunks.section present — schema OK."
-
-# ── 3. Confirmation gate (destructive) ───────────────────────────────────────
-echo ""
-echo "This TRUNCATEs papers CASCADE (papers/chunks/citations/formulas/figures/"
-echo "paper_identifiers) and re-ingests via ingest_all.sh. The DB content is"
-echo "rebuilt from scratch."
-read -r -p "Type 'MIGRATE' to proceed: " confirm
-[[ "$confirm" == "MIGRATE" ]] || { echo "Aborted — nothing changed."; exit 1; }
-
-# ── 4. Wipe ──────────────────────────────────────────────────────────────────
-echo "── TRUNCATE papers CASCADE ───────────────────────────────────"
-run_sql "TRUNCATE papers CASCADE"
-echo "   wiped."
-
-# ── 5. Re-ingest on canonical ids ────────────────────────────────────────────
-echo "── Re-ingest (ingest_all.sh) ─────────────────────────────────"
-bash "$CODEX_DIR/ingest_all.sh"
-
-# ── 6. AFTER snapshot + live verification ────────────────────────────────────
-echo "── AFTER migration ───────────────────────────────────────────"
-echo "   C-7 — url_form_ids should now be 0; sample ids should be bare:"
-run_sql "SELECT count(*) AS papers, count(*) FILTER (WHERE starts_with(id, 'https://')) AS url_form_ids FROM papers"
-run_sql "SELECT id FROM papers ORDER BY added_at LIMIT 3"
-
-# C-1 must be verified through the REAL resolver-based discovery_leads(), not a
-# raw cited_id check: cited_id/openalex_id stay in OpenAlex form, so the raw
-# "cited_id NOT IN papers.id" count is non-zero by design — the resolver is what
-# excludes ingested papers. Check that no ingested paper leaks into the leads.
-echo "   C-1 — real discovery_leads() must contain no already-ingested paper:"
-PYTHONPATH="$CODEX_DIR" "$PY" - <<'PYEOF'
-import os, psycopg
-from psycopg.rows import dict_row
-from codex.discover import discovery_leads
-try:
-    leads = discovery_leads(limit=100000)
-    with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row, connect_timeout=10) as c:
-        ingested = {r["id"] for r in c.execute("SELECT id FROM papers").fetchall()}
-        ingested |= {
-            r["openalex_id"]
-            for r in c.execute("SELECT openalex_id FROM papers WHERE openalex_id IS NOT NULL").fetchall()
-        }
-    leaked = sum(1 for lead in leads if lead["cited_id"] in ingested)
-    print(f"   leads={len(leads)}  ingested-papers-leaked-into-leads={leaked}")
-except Exception as e:
-    print(f"   DB ERROR: {type(e).__name__} (details suppressed)")
-PYEOF
-
-echo ""
-echo "✓ Migration complete."
-echo "  Expected: url_form_ids=0 (C-7 verified) and leaked=0 (C-1 verified) above."
-echo "  Next: re-run 'codex graph report' and refresh the ADR-F15 spike table (audit D-1)."

infra/schema.sql

@@ -13,7 +13,7 @@ CREATE EXTENSION IF NOT EXISTS vector;
 -- ---------------------------------------------------------------------
 -- Layer 1: Papers + full-text chunks
 -- ---------------------------------------------------------------------
-CREATE TABLE IF NOT EXISTS papers (
+CREATE TABLE papers (
     id            TEXT PRIMARY KEY,        -- canonical ID: arXiv ID or DOI
     openalex_id   TEXT UNIQUE,             -- e.g. W2741809807
     bibkey        TEXT UNIQUE,             -- BibTeX key -> .bib + Doxygen @cite
@@ -27,10 +27,10 @@ CREATE TABLE IF NOT EXISTS papers (
 );
 
 -- HNSW index for fast approximate nearest-neighbour search on abstracts.
-CREATE INDEX IF NOT EXISTS papers_abstract_emb_idx
+CREATE INDEX papers_abstract_emb_idx
     ON papers USING hnsw (abstract_emb vector_cosine_ops);
 
-CREATE TABLE IF NOT EXISTS chunks (
+CREATE TABLE chunks (
     id        BIGSERIAL PRIMARY KEY,
     paper_id  TEXT REFERENCES papers(id) ON DELETE CASCADE,
     ord       INT NOT NULL,               -- position within the paper
@@ -38,13 +38,13 @@ CREATE TABLE IF NOT EXISTS chunks (
     embedding vector(1024)
 );
 
-CREATE INDEX IF NOT EXISTS chunks_emb_idx
+CREATE INDEX chunks_emb_idx
     ON chunks USING hnsw (embedding vector_cosine_ops);
-CREATE INDEX IF NOT EXISTS chunks_paper_idx ON chunks (paper_id);
+CREATE INDEX chunks_paper_idx ON chunks (paper_id);
 
 -- Sparse / keyword hits for exact mathematical terminology (hybrid search).
 -- Dense (above) + full-text (below) combined = robust against math terms.
-CREATE INDEX IF NOT EXISTS chunks_fts_idx
+CREATE INDEX chunks_fts_idx
     ON chunks USING gin (to_tsvector('english', content));
 
 -- ---------------------------------------------------------------------
@@ -53,19 +53,19 @@ CREATE INDEX IF NOT EXISTS chunks_fts_idx
 --   edges to not-yet-ingested papers are intentionally preserved —
 --   those are your discovery leads.
 -- ---------------------------------------------------------------------
-CREATE TABLE IF NOT EXISTS citations (
+CREATE TABLE citations (
     citing_id TEXT REFERENCES papers(id) ON DELETE CASCADE,
     cited_id  TEXT NOT NULL,               -- arXiv/DOI/OpenAlex ID of the target
     context   TEXT,                        -- optional: citation context (S2)
     PRIMARY KEY (citing_id, cited_id)
 );
 
-CREATE INDEX IF NOT EXISTS citations_cited_idx ON citations (cited_id);
+CREATE INDEX citations_cited_idx ON citations (cited_id);
 
 -- ---------------------------------------------------------------------
 -- Layer 3: Provenance  (the "cleanly couple" goal)
 -- ---------------------------------------------------------------------
-CREATE TABLE IF NOT EXISTS code_links (
+CREATE TABLE code_links (
     id        BIGSERIAL PRIMARY KEY,
     symbol    TEXT NOT NULL,               -- e.g. 'dec::hodge_star' or 'file.cpp:120'
     paper_id  TEXT REFERENCES papers(id) ON DELETE SET NULL,
@@ -74,8 +74,8 @@ CREATE TABLE IF NOT EXISTS code_links (
     added_at  TIMESTAMPTZ DEFAULT now()
 );
 
-CREATE INDEX IF NOT EXISTS code_links_symbol_idx ON code_links (symbol);
+CREATE INDEX code_links_symbol_idx ON code_links (symbol);
-CREATE INDEX IF NOT EXISTS code_links_paper_idx  ON code_links (paper_id);
+CREATE INDEX code_links_paper_idx  ON code_links (paper_id);
 
 -- ---------------------------------------------------------------------
 -- Example query: discovery leads
@@ -88,58 +88,3 @@ CREATE INDEX IF NOT EXISTS code_links_paper_idx  ON code_links (paper_id);
 -- GROUP  BY cited_id
 -- ORDER  BY pull DESC
 -- LIMIT  20;
-
--- ---------------------------------------------------------------------
--- F-09 Rich Parsing: formulas + figures
--- ---------------------------------------------------------------------
-CREATE TABLE IF NOT EXISTS formulas (
-    id          BIGSERIAL PRIMARY KEY,
-    paper_id    TEXT REFERENCES papers(id) ON DELETE CASCADE,
-    page        INT,
-    raw_latex   TEXT NOT NULL,
-    context     TEXT,
-    eq_label    TEXT,
-    embedding   vector(1024)
-);
-
-CREATE INDEX IF NOT EXISTS formulas_emb_idx
-    ON formulas USING hnsw (embedding vector_cosine_ops);
-CREATE INDEX IF NOT EXISTS formulas_paper_idx ON formulas (paper_id);
-CREATE INDEX IF NOT EXISTS formulas_fts_idx
-    ON formulas USING gin (
-        to_tsvector('english', raw_latex || ' ' || coalesce(context, ''))
-    );
-
-CREATE TABLE IF NOT EXISTS figures (
-    id          BIGSERIAL PRIMARY KEY,
-    paper_id    TEXT REFERENCES papers(id) ON DELETE CASCADE,
-    page        INT,
-    caption     TEXT,
-    image_path  TEXT,
-    embedding   vector(1024)
-);
-
-CREATE INDEX IF NOT EXISTS figures_emb_idx
-    ON figures USING hnsw (embedding vector_cosine_ops);
-CREATE INDEX IF NOT EXISTS figures_paper_idx ON figures (paper_id);
-
--- F-16 Chunk Quality Gate: section classification
-ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT;
-
--- F-17 Paper identifier aliases: one paper → many OpenAlex IDs
--- (arXiv preprint ID + journal published ID + any future version)
--- Each OpenAlex ID maps to exactly one paper (UNIQUE on openalex_id).
--- Populated automatically by ingest; add extra aliases manually.
-CREATE TABLE IF NOT EXISTS paper_identifiers (
-    paper_id    TEXT NOT NULL REFERENCES papers(id) ON DELETE CASCADE,
-    openalex_id TEXT NOT NULL,
-    PRIMARY KEY (paper_id, openalex_id)
-);
-
-CREATE UNIQUE INDEX IF NOT EXISTS paper_identifiers_oa_idx
-    ON paper_identifiers (openalex_id);
-
--- Seed from existing papers.openalex_id
-INSERT INTO paper_identifiers (paper_id, openalex_id)
-SELECT id, openalex_id FROM papers WHERE openalex_id IS NOT NULL
-ON CONFLICT DO NOTHING;

ingest_all.sh

@@ -1,116 +0,0 @@
-#!/usr/bin/env bash
-# Batch ingest — writes directly to Jetson PostgreSQL via SSH tunnel.
-#
-# Prerequisites:
-#   ssh -f -N -L 5433:localhost:5432 alfred@jetson.local
-#   uv sync  (done once)
-#
-# Usage: bash ingest_all.sh 2>&1 | tee ingest_all.log
-set -euo pipefail
-
-PAPERS_DIR="/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt"
-CODEX_DIR="$(cd "$(dirname "$0")" && pwd)"
-ENV_FILE="$CODEX_DIR/.env.jetson-ingest"
-TUNNEL_PORT=5433
-
-# ── 1. Verify SSH tunnel is active ───────────────────────────────────────────
-if ! nc -z localhost "$TUNNEL_PORT" 2>/dev/null; then
-  echo "ERROR: SSH tunnel not active on port $TUNNEL_PORT"
-  echo "Run: ssh -f -N -L 5433:localhost:5432 alfred@jetson.local"
-  exit 1
-fi
-echo "✓ SSH tunnel active on port $TUNNEL_PORT"
-
-# ── 1b. Clear macOS UF_HIDDEN flag on editable-install .pth files ─────────────
-# Python 3.13 skips .pth files with this flag set (new in 3.13); uv sets it.
-# --no-sync below prevents uv from re-syncing (which would re-set the flag).
-SP="$CODEX_DIR/.venv/lib/python3.13/site-packages"
-for pth in "$SP"/_editable_impl_codex*.pth; do
-  [[ -f "$pth" ]] && chflags nohidden "$pth" 2>/dev/null || true
-done
-echo "✓ Cleared hidden flags on editable .pth files"
-
-# ── 2. Load env (DATABASE_URL → Jetson via tunnel) ───────────────────────────
-# pydantic-settings reads DATABASE_URL directly from the environment.
-# DOTENV_PATH is not supported; export each var explicitly.
-set -a; source "$ENV_FILE"; set +a
-export DATABASE_URL GROBID_URL OLLAMA_BASE_URL EMBEDDING_MODEL EMBEDDING_DIM OPENALEX_MAILTO
-echo "✓ DATABASE_URL: $DATABASE_URL"
-
-# ── 3. Paper ID → source file mapping ────────────────────────────────────────
-# Format: "PAPER_ID|filename.txt"
-# SKIP: farkas-kra-1992-riemann-surfaces.txt (textbook, no OpenAlex entry)
-declare -a PAPERS=(
-  # arXiv papers
-  "math/0603097|springborn-2008-weighted-delaunay-hyperideal.txt"
-  "1005.2698|bobenko-pinkall-springborn-2015-discrete-conformal-maps.txt"
-  "math/0306167|luo-2004-combinatorial-yamabe-flow.txt"
-  "math/0203250|bobenko-springborn-2004-circle-patterns.txt"
-  "math/0503219|bobenko-springborn-2007-discrete-laplace-beltrami.txt"
-  "2310.17529|bobenko-lutz-2025-non-euclidean-dce.txt"
-  "2305.10988|bobenko-lutz-2024-decorated-conformal-maps.txt"
-  "2206.13461|lutz-2023-canonical-tessellations.txt"
-  "2601.22903|bowers-bowers-lutz-2026-koebe-rigidity.txt"
-  "1911.00966|pinkall-springborn-2021-liouville.txt"
-  "1505.01341|born-bucking-springborn-2015-quasiconformal.txt"
-  "0906.1560|glickenstein-2011-discrete-conformal-variations.txt"
-  "math/0001176|rivin-schlenker-2000-schlafli-formula-arxiv-preprint.txt"
-  # DOI papers
-  "10.1007/s00454-019-00132-8|springborn-2020-ideal-hyperbolic-polyhedra.txt"
-  "10.1145/1964921.1964997|alexa-wardetzky-2011-discrete-laplacians-polygonal.txt"
-  "10.1111/cgf.13931|bunge-herholz-kazhdan-botsch-2020-polygon-laplacian.txt"
-  "10.1145/3450626.3459763|gillespie-springborn-crane-2021-discrete-conformal-equivalence.txt"
-  "10.1145/3306346.3323042|sharp-soliman-crane-2019-navigating-intrinsic-triangulations.txt"
-  "10.1145/3197517.3201367|soliman-slepcv-crane-2018-optimal-cone-singularities.txt"
-  "10.1145/3132705|sawhney-crane-2017-boundary-first-flattening.txt"
-  "10.1145/2767000|knoppel-crane-pinkall-schroder-2015-stripe-patterns.txt"
-  "10.5555/1070432.1070581|erickson-whittlesey-2005-greedy-homotopy-homology.txt"
-  "10.1145/1185657.1185665|desbrun-kanso-tong-2006-discrete-differential-forms.txt"
-  "10.1080/10586458.1993.10504266|pinkall-polthier-1993-computing-discrete-minimal-surfaces.txt"
-  "10.1007/s11040-021-09394-2|bobenko-bucking-2021-period-matrices.txt"
-  "10.14279/depositonce-20357|lutz-2024-thesis.txt"
-  "10.14279/depositonce-5415|sechelmann-2016-thesis.txt"
-  # Book chapters (txt contains full book; ID = cited chapter)
-  "10.1007/0-387-29555-0_13|precopa-molnar-eds-2006-non-euclidean-geometries-book.txt"
-  "10.1007/978-3-642-17413-1_7|bobenko-klein-eds-2011-computational-approach-riemann-surfaces-book.txt"
-)
-
-# ── 4. Ingest loop ────────────────────────────────────────────────────────────
-OK=0; FAIL=0; SKIP=0
-FAILED_IDS=()
-
-cd "$CODEX_DIR"
-
-for entry in "${PAPERS[@]}"; do
-  PAPER_ID="${entry%%|*}"
-  FILENAME="${entry##*|}"
-  SOURCE="$PAPERS_DIR/$FILENAME"
-
-  if [[ ! -f "$SOURCE" ]]; then
-    echo "SKIP (file missing): $FILENAME"
-    (( SKIP++ )) || true
-    continue
-  fi
-
-  echo ""
-  echo "── Ingesting: $PAPER_ID"
-  echo "   source: $FILENAME"
-
-  if PYTHONPATH="$CODEX_DIR" "$CODEX_DIR/.venv/bin/codex" ingest "$PAPER_ID" --source "$SOURCE"; then
-    (( OK++ )) || true
-  else
-    echo "FAILED: $PAPER_ID"
-    FAILED_IDS+=("$PAPER_ID")
-    (( FAIL++ )) || true
-  fi
-done
-
-# ── 5. Summary ────────────────────────────────────────────────────────────────
-echo ""
-echo "════════════════════════════════════════"
-echo "Ingest complete: $OK OK, $FAIL FAILED, $SKIP SKIPPED"
-if [[ ${#FAILED_IDS[@]} -gt 0 ]]; then
-  echo "Failed IDs:"
-  for id in "${FAILED_IDS[@]}"; do echo "  $id"; done
-fi
-echo "════════════════════════════════════════"

pyproject.toml

@@ -13,18 +13,13 @@ dependencies = [
     "pgvector>=0.3",
     "pydantic-settings>=2",
     "sentence-transformers>=3",
-    "flagembedding>=1.2",
     "typer>=0.12",
     "httpx>=0.27",
     "tenacity>=8",
-    "pymupdf>=1.24",
-    "pix2tex>=0.1.4",
-    "mcp[cli]>=1.0",
 ]
 
 [project.scripts]
 codex = "codex.cli:app"
-codex-mcp = "codex.mcp_server:main"
 
 [dependency-groups]
 dev = [

scripts/__init__.py

@@ -1 +0,0 @@
-"""Operational one-off scripts (importable for tests)."""

scripts/ra_grobid_backfill.py

@@ -1,227 +0,0 @@
-"""R-A — GROBID reference backfill (references-only, idempotent).
-
-Adds GROBID-parsed citations to the live corpus **without** re-chunking the body,
-so the DQ-3-verified-clean ``.txt`` chunks are preserved. Complements the DQ-1
-Semantic-Scholar supplement with references the APIs lack (notably the two
-TU-Berlin ``depositonce`` theses, which neither OpenAlex nor S2 indexes).
-
-Why references-only instead of ``ingest_paper(source_path=pdf)``
-----------------------------------------------------------------
-The PDF branch of :func:`codex.ingest.ingest_paper` runs Nougat OCR and then
-``DELETE FROM chunks`` + re-INSERT — it would replace the clean ``.txt`` chunks
-(DQ-3) with unmeasured PDF-OCR text and additionally requires the Nougat server.
-This backfill touches only the ``citations`` table (``ON CONFLICT DO NOTHING``),
-so it is safe, idempotent, and needs only a reachable GROBID server.
-
-Prerequisite
-------------
-A reachable GROBID server. The corpus' ``GROBID_URL`` points at the Jetson
-(``http://jetson.local:8070``). GROBID now runs natively on that aarch64 host
-via the multi-arch CRF-only image (``grobid/grobid:0.9.0-crf``) — see
-``docs/infra/grobid-jetson.md`` for the deploy + docker-permission fix and the
-``GET /api/isalive`` check. (The old ``grobid/grobid:0.8.2`` pin was amd64-only,
-which is why GROBID never came up; pass ``--grobid-url`` to override the default.)
-Use the SSH DB tunnel for ``DATABASE_URL`` as documented in
-``docs/audit/DATA-QUALITY-2026-06-15.md``.
-
-Usage
------
-    # dry run (no writes) over the two zero-edge theses
-    PYTHONPATH=. python scripts/ra_grobid_backfill.py --zero-edge-only
-
-    # write every paper's GROBID-parsed refs to the live DB
-    PYTHONPATH=. python scripts/ra_grobid_backfill.py --write
-
-Default is **dry-run**; pass ``--write`` to mutate the live DB.
-"""
-
-from __future__ import annotations
-
-import argparse
-import logging
-from pathlib import Path
-
-import psycopg
-import psycopg.rows
-
-from codex.config import get_settings
-from codex.db import get_conn
-from codex.models import Citation
-from codex.parsing.grobid import extract_references
-
-logger = logging.getLogger(__name__)
-
-DEFAULT_PDF_DIR = "/Users/tarikmoussa/Desktop/ConformalLabpp/papers"
-
-
-def _normalize_cited_id(doi: str = "", arxiv_id: str = "") -> str | None:
-    """Normalize a GROBID-extracted reference id to the corpus' canonical form.
-
-    DOIs → bare, lower-cased (strip ``https://doi.org/`` / ``http://`` / ``doi:``),
-    matching ``papers.id`` and ``codex.sources.openalex._normalize_doi`` (DQ-5).
-    arXiv ids → strip a leading ``arXiv:`` prefix, leaving the bare id
-    (``2305.10988``, ``math/0603097``). DOI takes precedence when both exist.
-    Kept self-contained rather than reusing ``ingest._norm_cited_id`` because it
-    additionally strips the ``arXiv:`` prefix (which GROBID can emit but that
-    helper leaves untouched), and to avoid importing a private cross-module name.
-    """
-    doi = (doi or "").strip()
-    if doi:
-        s = doi.lower()
-        for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
-            if s.startswith(prefix):
-                return s[len(prefix) :]
-        return s
-    arx = (arxiv_id or "").strip()
-    if arx:
-        return arx[len("arxiv:") :].strip() if arx.lower().startswith("arxiv:") else arx
-    return None
-
-
-def build_grobid_citations(
-    paper_id: str,
-    pdf_path: str | Path,
-    grobid_url: str | None = None,
-    timeout: float = 60.0,
-) -> list[Citation]:
-    """Extract a paper's references via GROBID and return normalized Citations.
-
-    ``citing_id`` is the canonical ``paper_id``; ``cited_id`` is normalized.
-    Self-citations and within-paper duplicates are dropped. Refs with neither a
-    DOI nor an arXiv id are skipped (they cannot join the graph).
-    """
-    refs = extract_references(str(pdf_path), grobid_url=grobid_url, timeout=timeout)
-    seen: set[str] = set()
-    citations: list[Citation] = []
-    for ref in refs:
-        cited = _normalize_cited_id(ref.get("doi", ""), ref.get("arxiv_id", ""))
-        if not cited or cited == paper_id or cited in seen:
-            continue
-        seen.add(cited)
-        citations.append(Citation(citing_id=paper_id, cited_id=cited))
-    return citations
-
-
-def _coverage(conn: psycopg.Connection[psycopg.rows.DictRow]) -> tuple[int, int, int]:
-    """Return (n_papers, n_papers_with_out_edges, n_citations) from the live DB."""
-    row = conn.execute(
-        """
-        SELECT (SELECT count(*) FROM papers)                    AS papers,
-               (SELECT count(DISTINCT citing_id) FROM citations) AS citing,
-               (SELECT count(*) FROM citations)                  AS edges
-        """
-    ).fetchone()
-    assert row is not None
-    return int(row["papers"]), int(row["citing"]), int(row["edges"])
-
-
-def main(argv: list[str] | None = None) -> int:
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--pdf-dir", default=DEFAULT_PDF_DIR, help="directory of source PDFs")
-    parser.add_argument("--grobid-url", default=None, help="override Settings().grobid_url")
-    parser.add_argument("--only", default=None, help="restrict to a single paper id")
-    parser.add_argument(
-        "--zero-edge-only",
-        action="store_true",
-        help="only papers with 0 out-edges (the R-A primary targets)",
-    )
-    parser.add_argument("--limit", type=int, default=None, help="cap number of papers")
-    parser.add_argument(
-        "--timeout",
-        type=float,
-        default=300.0,
-        help="GROBID HTTP timeout (s); theses can need >60s, esp. emulated",
-    )
-    parser.add_argument(
-        "--write", action="store_true", help="WRITE to the live DB (default: dry-run)"
-    )
-    args = parser.parse_args(argv)
-
-    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-    grobid_url = args.grobid_url or get_settings().grobid_url
-    pdf_dir = Path(args.pdf_dir)
-    pdfs = {p.stem: p for p in pdf_dir.glob("*.pdf")}
-
-    # Read the target list in a short-lived connection, then release it: the GROBID
-    # extraction below is a slow per-paper network loop and must NOT hold an
-    # idle-in-transaction connection open across it (a mid-loop SSH-tunnel drop would
-    # otherwise abort the whole batch — see docs/infra/jetson-ssh-stability.md).
-    with get_conn() as conn:
-        rows = conn.execute(
-            """
-            SELECT p.id, p.source_path,
-                   (SELECT count(*) FROM citations ci WHERE ci.citing_id = p.id) AS out_edges
-            FROM papers p
-            ORDER BY out_edges, p.id
-            """
-        ).fetchall()
-
-    targets: list[tuple[str, Path]] = []
-    for r in rows:
-        if args.only and r["id"] != args.only:
-            continue
-        if args.zero_edge_only and r["out_edges"] != 0:
-            continue
-        stem = Path(r["source_path"]).stem if r["source_path"] else None
-        pdf = pdfs.get(stem) if stem else None
-        if pdf is None:
-            logger.warning("no PDF for %s (stem=%s) — skipping", r["id"], stem)
-            continue
-        targets.append((r["id"], pdf))
-    if args.limit is not None:
-        targets = targets[: args.limit]
-
-    logger.info(
-        "GROBID=%s | papers to process=%d | mode=%s",
-        grobid_url,
-        len(targets),
-        "WRITE" if args.write else "DRY-RUN",
-    )
-
-    # GROBID extraction — no DB connection held during this slow network loop.
-    all_citations: list[Citation] = []
-    for paper_id, pdf in targets:
-        try:
-            cits = build_grobid_citations(
-                paper_id, pdf, grobid_url=grobid_url, timeout=args.timeout
-            )
-        except Exception as exc:
-            logger.warning("GROBID extraction failed for %s (%s): %s", paper_id, pdf.name, exc)
-            continue
-        logger.info("%-38s %3d refs  (%s)", paper_id, len(cits), pdf.name)
-        all_citations.extend(cits)
-
-    logger.info("total candidate citations: %d", len(all_citations))
-
-    if not args.write:
-        logger.info("DRY-RUN — no rows written. Re-run with --write to apply.")
-        return 0
-
-    # Fresh connection for the write — the read connection was already released.
-    with get_conn() as conn:
-        before = _coverage(conn)
-        with conn.cursor() as cur:
-            cur.executemany(
-                """
-                INSERT INTO citations (citing_id, cited_id, context)
-                VALUES (%s, %s, %s)
-                ON CONFLICT (citing_id, cited_id) DO NOTHING
-                """,
-                [(c.citing_id, c.cited_id, c.context) for c in all_citations],
-            )
-        conn.commit()
-        after = _coverage(conn)
-    logger.info(
-        "coverage papers=%d  citing %d->%d  edges %d->%d (+%d)",
-        after[0],
-        before[1],
-        after[1],
-        before[2],
-        after[2],
-        after[2] - before[2],
-    )
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scripts/rc_tex_reingest.py

@@ -1,131 +0,0 @@
-"""R-C — re-ingest arXiv papers from LaTeX source (section-aware, higher fidelity).
-
-For each arXiv paper, download its source via :func:`codex.sources.arxiv.fetch_source`
-(multi-file ``\\input``/``\\include`` flattened), write it to a temp ``.tex``, and
-re-ingest through :func:`codex.ingest.ingest_paper`. The ``.tex`` ingest path is
-section-aware (:func:`codex.parsing.tex.chunk_sections`), so the stored ``section``
-column is labelled from real ``\\section`` headings instead of mostly ``body`` —
-serving DQ-3 (fidelity vs the OCR'd ``.txt``) and audit R-12 (weak ``section`` column).
-
-Only arXiv papers have downloadable source; DOI/journal papers and books are skipped.
-Re-ingest REPLACES the paper's chunks (DELETE + re-INSERT) — that is the intended R-C
-improvement (flattened ``.txt`` → clean ``.tex`` chunks). It is idempotent and
-**dry-run by default**; pass ``--write`` to mutate the live corpus.
-
-Usage
------
-    # dry-run: list arXiv papers + source availability (no writes)
-    PYTHONPATH=. python scripts/rc_tex_reingest.py
-
-    # re-ingest every arXiv paper from .tex
-    PYTHONPATH=. python scripts/rc_tex_reingest.py --write
-
-    # one paper
-    PYTHONPATH=. python scripts/rc_tex_reingest.py --only 2305.10988 --write
-
-Needs ``DATABASE_URL`` (Jetson tunnel) as documented in
-``docs/audit/DATA-QUALITY-2026-06-15.md``.
-"""
-
-from __future__ import annotations
-
-import argparse
-import logging
-import tempfile
-
-import psycopg
-import psycopg.rows
-
-from codex.db import get_conn
-from codex.ingest import ingest_paper
-from codex.sources import arxiv
-
-logger = logging.getLogger(__name__)
-
-
-def is_arxiv_id(paper_id: str) -> bool:
-    """Return True for arXiv ids (the only papers with downloadable LaTeX source).
-
-    arXiv ids are modern (``2305.10988``) or legacy (``math/0603097``). DOIs start
-    with ``10.``; OpenAlex W-ids start with ``W`` or an ``https://openalex.org/`` URL.
-    """
-    p = (paper_id or "").strip()
-    # Exclude DOIs (incl. the arXiv DOI 10.48550/arXiv.*), OpenAlex W-ids, and any
-    # http(s) URL form; everything else is treated as a bare arXiv id.
-    return (
-        bool(p)
-        and not p.startswith(("10.", "W"))
-        and not p.lower().startswith(("http://", "https://"))
-    )
-
-
-def _section_dist(conn: psycopg.Connection[psycopg.rows.DictRow], paper_id: str) -> dict[str, int]:
-    """Return the section-label distribution for a paper's stored chunks."""
-    rows = conn.execute(
-        "SELECT coalesce(section, '(null)') AS s, count(*) AS n "
-        "FROM chunks WHERE paper_id = %s GROUP BY 1 ORDER BY 2 DESC",
-        (paper_id,),
-    ).fetchall()
-    return {r["s"]: int(r["n"]) for r in rows}
-
-
-def main(argv: list[str] | None = None) -> int:
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--only", default=None, help="restrict to a single arXiv id")
-    parser.add_argument("--limit", type=int, default=None, help="cap number of papers")
-    parser.add_argument(
-        "--write", action="store_true", help="re-ingest into the live DB (default: dry-run)"
-    )
-    args = parser.parse_args(argv)
-    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
-
-    with get_conn() as conn:
-        rows = conn.execute("SELECT id FROM papers ORDER BY id").fetchall()
-    targets = [
-        r["id"]
-        for r in rows
-        if is_arxiv_id(r["id"]) and (args.only is None or r["id"] == args.only)
-    ]
-    if args.limit is not None:
-        targets = targets[: args.limit]
-
-    logger.info(
-        "arXiv papers to process=%d | mode=%s",
-        len(targets),
-        "WRITE" if args.write else "DRY-RUN",
-    )
-
-    for pid in targets:
-        try:
-            src = arxiv.fetch_source(pid)
-        except Exception as exc:
-            logger.warning("%s: source fetch failed: %s", pid, exc)
-            continue
-        if not src:
-            logger.warning("%s: no LaTeX source available — skipping", pid)
-            continue
-        if not args.write:
-            logger.info("%s: source OK (%d chars)", pid, len(src))
-            continue
-
-        with get_conn() as conn:
-            before = _section_dist(conn, pid)
-        with tempfile.NamedTemporaryFile("w", suffix=".tex", delete=True) as tf:
-            tf.write(src)
-            tf.flush()
-            res = ingest_paper(pid, source_path=tf.name)
-        with get_conn() as conn:
-            after = _section_dist(conn, pid)
-        logger.info(
-            "%s: chunks %d->%d | section before=%s after=%s",
-            pid,
-            sum(before.values()),
-            res.chunks_upserted,
-            before,
-            after,
-        )
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

spike.py

@@ -0,0 +1,68 @@
+"""Spike: verify BGE-M3 dense+sparse encoding via sentence-transformers.
+
+RESULT: NO-GO.
+
+The session prompt for F-04 specified an `encode()` call signature of
+``encode([...], return_dense=True, return_sparse=True, batch_size=2)``.
+This signature originates from the ``FlagEmbedding.BGEM3FlagModel`` API,
+not ``sentence_transformers.SentenceTransformer``.
+
+With ``sentence-transformers==5.5.1`` and ``BAAI/bge-m3``:
+
+    >>> model.encode(['hello', 'world'], return_dense=True, return_sparse=True)
+    ValueError: SentenceTransformer.encode() has been called with additional
+    keyword arguments that this model does not use:
+    ['return_sparse', 'return_dense'].
+
+The vanilla call ``model.encode([...])`` returns dense vectors of shape
+``(2, 1024)`` and dtype ``float32`` as expected — dense embeddings work.
+Sparse / lexical-weights output is NOT exposed via the standard
+``SentenceTransformer`` wrapper.  To get the BGE-M3 sparse head we must:
+
+    (a) Install ``FlagEmbedding`` and load via ``BGEM3FlagModel``, or
+    (b) Drive the underlying ``transformers`` model directly and apply the
+        sparse projection head ourselves, or
+    (c) Use ``sentence_transformers.SparseEncoder`` with a different
+        (SPLADE-style) checkpoint.
+
+Decision required from Theorist/Leader: which path?
+"""
+
+from __future__ import annotations
+
+import sys
+
+
+def main() -> int:
+    try:
+        from sentence_transformers import SentenceTransformer
+
+        model = SentenceTransformer("BAAI/bge-m3")
+        out = model.encode(
+            ["hello", "world"],
+            return_dense=True,
+            return_sparse=True,
+            batch_size=2,
+        )
+
+        if isinstance(out, dict):
+            dense = out.get("dense_vecs")
+            sparse = out.get("lexical_weights")
+        else:
+            dense = out
+            sparse = None
+
+        assert dense is not None, "no dense output"
+        assert dense.shape == (2, 1024), f"dense shape {dense.shape} != (2, 1024)"
+        assert isinstance(sparse, list) and len(sparse) == 2, "sparse not list of 2"
+        assert all(isinstance(d, dict) for d in sparse), "sparse elements not dicts"
+    except Exception as exc:  # noqa: BLE001 — spike must catch everything
+        print(f"Spike: NO-GO — {type(exc).__name__}: {exc}")
+        return 1
+
+    print("Spike: GO")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

spike/F13_synthesis_spike.py

@@ -1,141 +0,0 @@
-"""F-13 synthesis spike — quick live-infra evaluation (WEGWERF-CODE).
-
-NOT a unit test. Not maintained. Run by hand against live infrastructure:
-
-  DB:    postgresql://researcher:change_me@localhost:5432/papers (podman, Mac)
-  LLM:   http://jetson.local:11434 (Jetson, qwen2.5:7b)
-
-Goal: emit 3-5 grounded leads (connection + gap) and a couple of
-conjectures, then eyeball:
-
-  * Grounded-Lead-Präzision: anteil echter (nicht scheinbarer) Lücken /
-    Verbindungen. Ziel ≥ 60 %.
-  * Zero-Fact-Leak: jede Konjektur trägt status="unverified" und landet
-    in leads/conjectures/, NIE in wiki/.
-
-Result is dumped to ``spike/spike_output.json`` and stdout.
-"""
-
-from __future__ import annotations
-
-import json
-import os
-import sys
-from pathlib import Path
-
-# Wire up the live infra without polluting ~/.env
-os.environ.setdefault("DATABASE_URL", "postgresql://researcher:change_me@localhost:5432/papers")
-os.environ.setdefault("OLLAMA_BASE_URL", "http://jetson.local:11434")
-os.environ.setdefault("SYNTHESIS_LLM_MODEL", "qwen2.5:7b")
-os.environ.setdefault("SYNTHESIS_TOP_K", "12")
-
-ROOT = Path(__file__).resolve().parent.parent
-sys.path.insert(0, str(ROOT))
-
-
-def _probe_db() -> bool:
-    try:
-        import psycopg
-
-        with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
-            n = conn.execute("SELECT count(*) FROM papers").fetchone()
-            print(f"[db ok] papers={n[0] if n else '?'}")
-            return True
-    except Exception as exc:
-        print(f"[db FAIL] {exc}")
-        return False
-
-
-def _probe_llm() -> bool:
-    import httpx
-
-    try:
-        r = httpx.get(os.environ["OLLAMA_BASE_URL"] + "/api/tags", timeout=3.0)
-        r.raise_for_status()
-        print(f"[llm ok] models={len(r.json().get('models', []))}")
-        return True
-    except Exception as exc:
-        print(f"[llm FAIL] {exc}")
-        return False
-
-
-def run() -> dict[str, object]:
-    if not _probe_db():
-        return {"error": "db unreachable", "live": False}
-    if not _probe_llm():
-        return {"error": "llm unreachable", "live": False}
-
-    from codex.config import get_settings  # noqa: E402  (after env wiring)
-    from codex.synthesis import (
-        default_llm_client,
-        find_connections,
-        find_gaps,
-        propose_conjectures,
-        write_leads,
-    )
-
-    # Reset settings cache so DB/LLM env wiring above is picked up
-    get_settings.cache_clear()
-    settings = get_settings()
-    print(f"[settings] top_k={settings.synthesis_top_k} model={settings.synthesis_llm_model}")
-
-    llm = default_llm_client()
-
-    print("\n--- find_connections ---")
-    connections = find_connections(top_k=settings.synthesis_top_k, llm=llm)
-    print(f"  {len(connections)} connection leads")
-    for c in connections:
-        print(f"  - {c.id}: {c.title}  (conf={c.confidence:.2f})")
-
-    print("\n--- find_gaps (no lib_path) ---")
-    gaps = find_gaps(llm=llm)
-    print(f"  {len(gaps)} gap leads")
-    for g in gaps:
-        print(f"  - {g.id}: {g.title}  (conf={g.confidence:.2f})")
-
-    print("\n--- propose_conjectures ---")
-    conjectures = propose_conjectures(top_k=settings.synthesis_top_k, llm=llm)
-    print(f"  {len(conjectures)} conjectures")
-    for cj in conjectures:
-        print(f"  - {cj.id}: {cj.title}")
-        print(f"      status={cj.status}  validation={cj.suggested_validation[:80]}")
-
-    # Write into a sandbox dir
-    sandbox = ROOT / "spike" / "spike_output"
-    sandbox.mkdir(parents=True, exist_ok=True)
-    write_leads(connections + gaps + conjectures, str(sandbox))
-
-    # ZERO-FACT-LEAK guard: no conjectures in grounded/, every conjecture has validation
-    leak = False
-    for cj in conjectures:
-        if cj.kind != "conjecture":
-            leak = True
-            print(f"[LEAK] {cj.id} kind={cj.kind}")
-        if cj.status != "unverified":
-            leak = True
-            print(f"[LEAK] {cj.id} status={cj.status} (expected unverified)")
-        if not cj.suggested_validation:
-            leak = True
-            print(f"[LEAK] {cj.id} missing suggested_validation")
-        if not cj.provenance:
-            leak = True
-            print(f"[LEAK] {cj.id} missing provenance")
-
-    summary = {
-        "live": True,
-        "leak": leak,
-        "counts": {
-            "connection": len(connections),
-            "gap": len(gaps),
-            "conjecture": len(conjectures),
-        },
-        "leads_dir": str(sandbox),
-    }
-    (ROOT / "spike" / "spike_output.json").write_text(json.dumps(summary, indent=2))
-    print("\n=== SUMMARY ===")
-    print(json.dumps(summary, indent=2))
-    return summary
-
-
-if __name__ == "__main__":
-    run()

spike/spike_output.json

@@ -1,10 +0,0 @@
-{
-  "live": true,
-  "leak": false,
-  "counts": {
-    "connection": 0,
-    "gap": 0,
-    "conjecture": 0
-  },
-  "leads_dir": "/Users/tarikmoussa/Desktop/codex-py/spike/spike_output"
-}

spike/spike_output/INDEX.md

@@ -1,11 +0,0 @@
-# Lead Index
-
-_Last updated 2026-06-14 06:35 UTC by `codex synthesis`_
-
-## Grounded leads (connection / gap / improvement)
-
-_None._
-
-## Conjectures (UNVERIFIED — quarantined here, never in wiki/)
-
-_None._

tests/cli/test_cli.py

@@ -1,384 +0,0 @@
-"""Tests for codex.cli — Typer command-line interface.
-
-All external calls (ingest, DB, embed, discovery, provenance) are mocked
-so the suite runs offline without real API calls.
-"""
-
-from __future__ import annotations
-
-from datetime import datetime
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-import numpy as np
-from typer.testing import CliRunner
-
-from codex.cli import app
-from codex.ingest import IngestResult
-from codex.models import CodeLink
-
-runner = CliRunner()
-
-
-class TestIngest:
-    """Tests for `codex ingest` command."""
-
-    def test_ingest_command(self) -> None:
-        """Test basic ingest command with mocked ingest_paper."""
-        with patch("codex.ingest.ingest_paper") as mock_ingest:
-            mock_ingest.return_value = IngestResult(
-                paper_id="2301.07041",
-                chunks_upserted=5,
-                citations_upserted=3,
-            )
-
-            result = runner.invoke(app, ["ingest", "2301.07041"])
-
-            assert result.exit_code == 0
-            assert "2301.07041" in result.stdout
-            assert "5 chunks" in result.stdout
-            assert "3 citations" in result.stdout
-
-    def test_ingest_with_rich_flag(self) -> None:
-        """Test `ingest --rich` passes rich=True and reports formulas/figures."""
-        with patch("codex.ingest.ingest_paper") as mock_ingest:
-            mock_ingest.return_value = IngestResult(
-                paper_id="2301.07041",
-                chunks_upserted=5,
-                citations_upserted=3,
-                formulas_upserted=12,
-                figures_upserted=4,
-            )
-
-            result = runner.invoke(app, ["ingest", "2301.07041", "--source", "paper.pdf", "--rich"])
-
-            assert result.exit_code == 0
-            assert "12 formulas" in result.stdout
-            assert "4 figures" in result.stdout
-            # Verify rich=True was passed
-            _, kwargs = mock_ingest.call_args
-            assert kwargs.get("rich") is True or mock_ingest.call_args[1].get("rich") is True
-
-
-class TestSearch:
-    """Tests for `codex search` subcommands (paper + formula)."""
-
-    def test_search_command(self) -> None:
-        """Alias: delegates to test_search_paper_command for backwards compat."""
-        self.test_search_paper_command()
-
-    def test_search_paper_command(self) -> None:
-        """Test `search paper` command with mocked embedder and DB."""
-        with (
-            patch("codex.embed.get_embedder") as mock_embedder,
-            patch("codex.db.get_conn") as mock_get_conn,
-        ):
-            # Mock embedder - return numpy array with one row
-            mock_emb_instance = MagicMock()
-            mock_emb_instance.encode_dense.return_value = np.array([[0.1, 0.2, 0.3]])
-            mock_embedder.return_value = mock_emb_instance
-
-            # Mock DB connection
-            mock_conn = MagicMock()
-            mock_conn.__enter__.return_value = mock_conn
-            mock_conn.__exit__.return_value = None
-            mock_conn.execute.return_value.fetchall.return_value = [
-                {
-                    "id": "2301.07041",
-                    "title": "Test Paper",
-                    "year": 2023,
-                    "distance": 0.123,
-                },
-                {
-                    "id": "2301.07042",
-                    "title": "Another Paper",
-                    "year": 2023,
-                    "distance": 0.456,
-                },
-            ]
-            mock_get_conn.return_value = mock_conn
-
-            result = runner.invoke(app, ["search", "paper", "machine learning", "--limit", "2"])
-
-            assert result.exit_code == 0
-            assert "2301.07041" in result.stdout
-            assert "2301.07042" in result.stdout
-            assert "Test Paper" in result.stdout
-            assert "0.123" in result.stdout
-
-            # Regression guard (DQ-4): the pgvector `<->` operator requires the
-            # query embedding cast to ::vector. Without it, real Postgres raises
-            # "operator does not exist: vector <-> double precision[]". Mocked DBs
-            # don't catch the type error, so assert the cast is in the SQL.
-            executed_sql = mock_conn.execute.call_args_list[0][0][0]
-            assert "<->" in executed_sql and "::vector" in executed_sql
-
-    def test_search_formula_command(self) -> None:
-        """Test `search formula` command with mocked DB."""
-        with patch("codex.db.get_conn") as mock_get_conn:
-            mock_conn = MagicMock()
-            mock_conn.__enter__.return_value = mock_conn
-            mock_conn.__exit__.return_value = None
-            mock_conn.execute.return_value.fetchall.return_value = [
-                {
-                    "paper_id": "2301.07041",
-                    "page": 3,
-                    "raw_latex": r"\sum_{i=1}^{n} x_i",
-                    "context": "summation formula",
-                    "rank": 0.856,
-                },
-            ]
-            mock_get_conn.return_value = mock_conn
-
-            result = runner.invoke(app, ["search", "formula", "summation", "--limit", "5"])
-
-            assert result.exit_code == 0
-            assert "2301.07041" in result.stdout
-            assert "p.3" in result.stdout
-
-    def test_search_formula_no_results(self) -> None:
-        """`search formula` with no DB matches prints a 'no results' message."""
-        with patch("codex.db.get_conn") as mock_get_conn:
-            mock_conn = MagicMock()
-            mock_conn.__enter__.return_value = mock_conn
-            mock_conn.__exit__.return_value = None
-            mock_conn.execute.return_value.fetchall.return_value = []
-            mock_get_conn.return_value = mock_conn
-
-            result = runner.invoke(app, ["search", "formula", "nonexistent"])
-
-            assert result.exit_code == 0
-            assert "No matching" in result.stdout
-
-
-class TestDiscoverLeads:
-    """Tests for `codex discover leads` command."""
-
-    def test_discover_leads_command(self) -> None:
-        """Test discover leads command with mocked discovery_leads."""
-        with patch("codex.discover.discovery_leads") as mock_leads:
-            mock_leads.return_value = [
-                {"cited_id": "2301.07041", "pull": 5},
-                {"cited_id": "2301.07042", "pull": 3},
-            ]
-
-            result = runner.invoke(app, ["discover", "leads", "--limit", "2"])
-
-            assert result.exit_code == 0
-            assert "2301.07041" in result.stdout
-            assert "2301.07042" in result.stdout
-            assert "5×" in result.stdout
-            assert "3×" in result.stdout
-
-
-class TestDiscoverCiting:
-    """Tests for `codex discover citing` command."""
-
-    def test_discover_citing_command(self) -> None:
-        """Test discover citing command with mocked citing_papers."""
-        with patch("codex.discover.citing_papers") as mock_citing:
-            mock_citing.return_value = ["2301.07041", "2301.07042"]
-
-            result = runner.invoke(app, ["discover", "citing", "2301.07043"])
-
-            assert result.exit_code == 0
-            assert "2301.07041" in result.stdout
-            assert "2301.07042" in result.stdout
-
-
-class TestDiscoverCitedBy:
-    """Tests for `codex discover cited-by` command."""
-
-    def test_discover_cited_by_command(self) -> None:
-        """Test discover cited-by command with mocked cited_by."""
-        with patch("codex.discover.cited_by") as mock_cited:
-            mock_cited.return_value = ["2301.07041", "2301.07042"]
-
-            result = runner.invoke(app, ["discover", "cited-by", "2301.07043"])
-
-            assert result.exit_code == 0
-            assert "2301.07041" in result.stdout
-            assert "2301.07042" in result.stdout
-
-
-class TestDiscoverCocited:
-    """Tests for `codex discover cocited` command."""
-
-    def test_discover_cocited_command(self) -> None:
-        """Test discover cocited command with mocked cocited_papers."""
-        with patch("codex.discover.cocited_papers") as mock_cocited:
-            mock_cocited.return_value = [
-                {"paper_id": "2301.07041", "co_citations": 5},
-                {"paper_id": "2301.07042", "co_citations": 3},
-            ]
-
-            result = runner.invoke(app, ["discover", "cocited", "2301.07043"])
-
-            assert result.exit_code == 0
-            assert "2301.07041" in result.stdout
-            assert "2301.07042" in result.stdout
-            assert "5×" in result.stdout
-            assert "3×" in result.stdout
-
-
-class TestProvScan:
-    """Tests for `codex provenance scan` command."""
-
-    def test_prov_scan_command(self) -> None:
-        """Test provenance scan command with mocked scan_cite_tags."""
-        with patch("codex.provenance.scan_cite_tags") as mock_scan:
-            mock_scan.return_value = [
-                {
-                    "file": "src/main.cpp",
-                    "line": "42",
-                    "symbol": "compute_embedding",
-                    "bibkey": "2301.07041",
-                },
-                {
-                    "file": "src/utils.h",
-                    "line": "10",
-                    "symbol": "helper_func",
-                    "bibkey": "2301.07042",
-                },
-            ]
-
-            result = runner.invoke(app, ["provenance", "scan", "/path/to/src"])
-
-            assert result.exit_code == 0
-            assert "src/main.cpp" in result.stdout
-            assert "2301.07041" in result.stdout
-            assert "compute_embedding" in result.stdout
-
-
-class TestProvAddLink:
-    """Tests for `codex provenance add-link` command."""
-
-    def test_prov_add_link_command(self) -> None:
-        """Test provenance add-link command with mocked add_code_link."""
-        with patch("codex.provenance.add_code_link") as mock_add:
-            mock_link = CodeLink(
-                id=1,
-                symbol="my_func",
-                paper_id="2301.07041",
-                role="core_algorithm",
-                note="Primary reference",
-                added_at=datetime(2023, 1, 1),
-            )
-            mock_add.return_value = mock_link
-
-            result = runner.invoke(
-                app,
-                [
-                    "provenance",
-                    "add-link",
-                    "my_func",
-                    "--paper-id",
-                    "2301.07041",
-                    "--role",
-                    "core_algorithm",
-                    "--note",
-                    "Primary reference",
-                ],
-            )
-
-            assert result.exit_code == 0
-            assert "Created code_link" in result.stdout
-            assert "my_func" in result.stdout
-            assert "2301.07041" in result.stdout
-
-
-class TestProvLinks:
-    """Tests for `codex provenance links` command."""
-
-    def test_prov_links_command(self) -> None:
-        """Test provenance links command with mocked list_code_links."""
-        with patch("codex.provenance.list_code_links") as mock_links:
-            mock_links.return_value = [
-                CodeLink(
-                    id=1,
-                    symbol="my_func",
-                    paper_id="2301.07041",
-                    role="core",
-                    note=None,
-                    added_at=datetime(2023, 1, 1),
-                ),
-                CodeLink(
-                    id=2,
-                    symbol="another_func",
-                    paper_id="2301.07042",
-                    role="utility",
-                    note=None,
-                    added_at=datetime(2023, 1, 2),
-                ),
-            ]
-
-            result = runner.invoke(app, ["provenance", "links"])
-
-            assert result.exit_code == 0
-            assert "[1]" in result.stdout
-            assert "[2]" in result.stdout
-            assert "my_func" in result.stdout
-            assert "another_func" in result.stdout
-
-
-class TestProvBib:
-    """Tests for `codex provenance bib` command."""
-
-    def test_prov_bib_command(self) -> None:
-        """Test provenance bib command with mocked export_bib."""
-        with patch("codex.provenance.export_bib") as mock_export:
-            mock_export.return_value = "@article{2301.07041,\n  title = {Test},\n}"
-
-            result = runner.invoke(app, ["provenance", "bib"])
-
-            assert result.exit_code == 0
-            assert "@article" in result.stdout
-            assert "2301.07041" in result.stdout
-
-    def test_prov_bib_with_output_file(self, tmp_path: Any) -> None:
-        """Test provenance bib command writing to file."""
-        with patch("codex.provenance.export_bib") as mock_export:
-            mock_export.return_value = "@article{2301.07041,\n  title = {Test},\n}"
-
-            output_file = tmp_path / "refs.bib"
-            result = runner.invoke(app, ["provenance", "bib", "--output", str(output_file)])
-
-            assert result.exit_code == 0
-            assert "Wrote" in result.stdout
-            assert output_file.exists()
-            assert "@article" in output_file.read_text()
-
-
-class TestAsk:
-    """Tests for `codex ask` command."""
-
-    def test_ask_command_exits_1(self) -> None:
-        """Test that ask command exits with code 1 (not yet implemented)."""
-        result = runner.invoke(app, ["ask", "what is machine learning?"])
-
-        assert result.exit_code == 1
-        assert "not yet implemented" in result.stdout or "not yet implemented" in result.stderr
-
-
-# ---------------------------------------------------------------------------
-# D-05a/b — Entry-point smoke tests (python -m codex, no uv run)
-# ---------------------------------------------------------------------------
-
-
-class TestEntryPoint:
-    """Verify that `python -m codex` works without `uv run` (D-05a/b)."""
-
-    def test_python_m_codex_help(self) -> None:
-        """`python -m codex --help` exits 0 and prints usage."""
-        import subprocess
-        import sys
-
-        proc = subprocess.run(
-            [sys.executable, "-m", "codex", "--help"],
-            capture_output=True,
-            text=True,
-        )
-        assert proc.returncode == 0, f"Non-zero exit: {proc.stderr}"
-        assert "Usage" in proc.stdout or "usage" in proc.stdout.lower(), (
-            f"No usage in output: {proc.stdout[:200]}"
-        )

tests/discover/test_discover.py

@@ -1,145 +0,0 @@
-"""Tests for codex.discover — graph-based discovery queries.
-
-All DB access is mocked; no real PostgreSQL server required.
-"""
-
-from __future__ import annotations
-
-from contextlib import contextmanager
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _make_conn_cm(mock_conn: MagicMock) -> Any:
-    """Return a context-manager factory that yields *mock_conn*."""
-
-    @contextmanager  # type: ignore[arg-type]
-    def _cm() -> Any:
-        yield mock_conn
-
-    return _cm
-
-
-def _dict_row(**kwargs: Any) -> dict[str, Any]:
-    return dict(kwargs)
-
-
-# ---------------------------------------------------------------------------
-# 1. test_discovery_leads_returns_correct_format
-# ---------------------------------------------------------------------------
-
-
-def test_discovery_leads_returns_correct_format() -> None:
-    """Mock conn returns rows; verify output has correct shape and ordering."""
-    from codex.discover import discovery_leads
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _dict_row(cited_id="W111", pull=5),
-        _dict_row(cited_id="W222", pull=3),
-        _dict_row(cited_id="W333", pull=1),
-    ]
-
-    with patch("codex.discover.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        result = discovery_leads(limit=10)
-
-    assert len(result) == 3
-    assert result[0] == {"cited_id": "W111", "pull": 5}
-    assert result[1] == {"cited_id": "W222", "pull": 3}
-    assert result[2] == {"cited_id": "W333", "pull": 1}
-
-    # Verify the SQL and limit param were passed
-    call_args = mock_conn.execute.call_args
-    sql: str = call_args[0][0]
-    params: dict[str, Any] = call_args[0][1]
-    assert "cited_id NOT IN" in sql
-    assert "pull DESC" in sql
-    assert params["limit"] == 10
-
-
-# ---------------------------------------------------------------------------
-# 2. test_citing_papers
-# ---------------------------------------------------------------------------
-
-
-def test_citing_papers() -> None:
-    """Mock returns citing_id list; verify flat list of IDs is returned."""
-    from codex.discover import citing_papers
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _dict_row(citing_id="2301.00001"),
-        _dict_row(citing_id="2302.00002"),
-    ]
-
-    with patch("codex.discover.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        result = citing_papers("W999")
-
-    assert result == ["2301.00001", "2302.00002"]
-
-    call_args = mock_conn.execute.call_args
-    sql: str = call_args[0][0]
-    params: dict[str, Any] = call_args[0][1]
-    assert "cited_id" in sql
-    assert params["paper_id"] == "W999"
-
-
-# ---------------------------------------------------------------------------
-# 3. test_cited_by
-# ---------------------------------------------------------------------------
-
-
-def test_cited_by() -> None:
-    """Mock returns cited_id list; verify flat list of IDs is returned."""
-    from codex.discover import cited_by
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _dict_row(cited_id="10.1000/ref_a"),
-        _dict_row(cited_id="10.1000/ref_b"),
-    ]
-
-    with patch("codex.discover.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        result = cited_by("2301.00001")
-
-    assert result == ["10.1000/ref_a", "10.1000/ref_b"]
-
-    call_args = mock_conn.execute.call_args
-    sql: str = call_args[0][0]
-    params: dict[str, Any] = call_args[0][1]
-    assert "citing_id" in sql
-    assert params["paper_id"] == "2301.00001"
-
-
-# ---------------------------------------------------------------------------
-# 4. test_cocited_papers
-# ---------------------------------------------------------------------------
-
-
-def test_cocited_papers() -> None:
-    """Mock returns co-citation list; verify shape and ordering."""
-    from codex.discover import cocited_papers
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _dict_row(paper_id="W444", co_citations=7),
-        _dict_row(paper_id="W555", co_citations=2),
-    ]
-
-    with patch("codex.discover.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        result = cocited_papers("W111", limit=5)
-
-    assert len(result) == 2
-    assert result[0] == {"paper_id": "W444", "co_citations": 7}
-    assert result[1] == {"paper_id": "W555", "co_citations": 2}
-
-    call_args = mock_conn.execute.call_args
-    sql: str = call_args[0][0]
-    params: dict[str, Any] = call_args[0][1]
-    assert "co_citations DESC" in sql
-    assert params["paper_id"] == "W111"
-    assert params["limit"] == 5

tests/embed/test_embedder.py

@@ -1,186 +0,0 @@
-"""Tests for codex.embed.
-
-The real :class:`FlagEmbedding.BGEM3FlagModel` would download 2 GB of
-weights and load them onto the device.  We mock it out so the suite
-runs offline in milliseconds.
-"""
-
-from __future__ import annotations
-
-from typing import Any
-
-import numpy as np
-import pytest
-
-import codex.embed as embed_module
-from codex.embed import Embedder, get_embedder
-
-# ---------------------------------------------------------------------------
-# Fakes / fixtures
-# ---------------------------------------------------------------------------
-
-
-class _FakeBGEModel:
-    """Stand-in for :class:`BGEM3FlagModel` that records calls.
-
-    ``encode()`` returns deterministic fake vectors so the L2-norm
-    assertions are exact.  The ``call_count`` attribute lets tests
-    verify the single-forward-pass invariant of :meth:`Embedder.encode`.
-    """
-
-    DIM = 1024
-
-    def __init__(self, *args: Any, **kwargs: Any) -> None:
-        self.init_args = args
-        self.init_kwargs = kwargs
-        self.call_count = 0
-        self.last_kwargs: dict[str, Any] | None = None
-
-    def encode(
-        self,
-        sentences: list[str],
-        **kwargs: Any,
-    ) -> dict[str, Any]:
-        self.call_count += 1
-        self.last_kwargs = kwargs
-        n = len(sentences)
-        rng = np.random.default_rng(seed=42)
-        return {
-            "dense_vecs": rng.random((n, self.DIM)).astype(np.float32),
-            "lexical_weights": [{0: 0.5, 7: 0.25} for _ in range(n)],
-        }
-
-
-@pytest.fixture
-def fake_model(monkeypatch: pytest.MonkeyPatch) -> type[_FakeBGEModel]:
-    """Replace ``BGEM3FlagModel`` in ``codex.embed`` with the fake class."""
-    monkeypatch.setattr(embed_module, "BGEM3FlagModel", _FakeBGEModel)
-    return _FakeBGEModel
-
-
-@pytest.fixture(autouse=True)
-def reset_singleton() -> None:
-    """Reset the module-level singleton between tests."""
-    embed_module._embedder = None
-
-
-# ---------------------------------------------------------------------------
-# encode_dense
-# ---------------------------------------------------------------------------
-
-
-def test_encode_dense_shape_dtype_and_norm(fake_model: type[_FakeBGEModel]) -> None:
-    """Dense output is (N, dim) float32 with unit L2 rows."""
-    e = Embedder()
-    out = e.encode_dense(["a", "b"])
-
-    assert out.shape == (2, 1024)
-    assert out.dtype == np.float32
-    norms = np.linalg.norm(out, axis=1)
-    np.testing.assert_allclose(norms, [1.0, 1.0], atol=1e-5)
-
-
-def test_encode_dense_empty_input_returns_empty_matrix(
-    fake_model: type[_FakeBGEModel],
-) -> None:
-    """Empty input -> (0, dim) without invoking the model."""
-    e = Embedder()
-    # Reach into the wrapped model to verify it is not called.
-    fake = e._model
-    assert isinstance(fake, _FakeBGEModel)
-
-    out = e.encode_dense([])
-
-    assert out.shape == (0, 1024)
-    assert out.dtype == np.float32
-    assert fake.call_count == 0
-
-
-# ---------------------------------------------------------------------------
-# encode_sparse
-# ---------------------------------------------------------------------------
-
-
-def test_encode_sparse_returns_list_of_dicts(fake_model: type[_FakeBGEModel]) -> None:
-    """Sparse output has one dict per input string."""
-    e = Embedder()
-    out = e.encode_sparse(["a", "b"])
-
-    assert len(out) == 2
-    for row in out:
-        assert isinstance(row, dict)
-        for key, value in row.items():
-            assert isinstance(key, int)
-            assert isinstance(value, float)
-
-
-def test_encode_sparse_empty_input_returns_empty_list(
-    fake_model: type[_FakeBGEModel],
-) -> None:
-    """Empty input -> [] without invoking the model."""
-    e = Embedder()
-    fake = e._model
-    assert isinstance(fake, _FakeBGEModel)
-
-    out = e.encode_sparse([])
-
-    assert out == []
-    assert fake.call_count == 0
-
-
-# ---------------------------------------------------------------------------
-# encode (combined)
-# ---------------------------------------------------------------------------
-
-
-def test_encode_returns_tuple_and_uses_single_forward_pass(
-    fake_model: type[_FakeBGEModel],
-) -> None:
-    """encode() must issue exactly one model.encode() call for efficiency."""
-    e = Embedder()
-    fake = e._model
-    assert isinstance(fake, _FakeBGEModel)
-
-    dense, sparse = e.encode(["a"])
-
-    assert isinstance(dense, np.ndarray)
-    assert dense.shape == (1, 1024)
-    assert dense.dtype == np.float32
-    np.testing.assert_allclose(np.linalg.norm(dense, axis=1), [1.0], atol=1e-5)
-
-    assert isinstance(sparse, list)
-    assert len(sparse) == 1
-    assert isinstance(sparse[0], dict)
-
-    assert fake.call_count == 1
-    assert fake.last_kwargs is not None
-    assert fake.last_kwargs.get("return_dense") is True
-    assert fake.last_kwargs.get("return_sparse") is True
-
-
-def test_encode_empty_input(fake_model: type[_FakeBGEModel]) -> None:
-    """Empty input -> ((0, dim), [])."""
-    e = Embedder()
-    fake = e._model
-    assert isinstance(fake, _FakeBGEModel)
-
-    dense, sparse = e.encode([])
-
-    assert dense.shape == (0, 1024)
-    assert dense.dtype == np.float32
-    assert sparse == []
-    assert fake.call_count == 0
-
-
-# ---------------------------------------------------------------------------
-# Singleton
-# ---------------------------------------------------------------------------
-
-
-def test_get_embedder_returns_singleton(fake_model: type[_FakeBGEModel]) -> None:
-    """Two calls return the same object."""
-    first = get_embedder()
-    second = get_embedder()
-
-    assert first is second
-    assert isinstance(first, Embedder)

tests/graph/test_cli.py

@@ -1,256 +0,0 @@
-"""CLI tests for `codex graph` sub-commands (F-15)."""
-
-from __future__ import annotations
-
-from contextlib import contextmanager
-from unittest.mock import MagicMock, patch
-
-import networkx as nx
-import numpy as np
-from typer.testing import CliRunner
-
-from codex.cli import app
-
-runner = CliRunner()
-
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _make_conn_cm(conn: MagicMock):
-    @contextmanager
-    def _cm():
-        yield conn
-
-    return _cm
-
-
-def _make_conn_with_paper_ids(*paper_ids: str) -> MagicMock:
-    """Mock conn that returns paper rows for SELECT id, title FROM papers."""
-    conn = MagicMock()
-    conn.execute.return_value.fetchall.return_value = [
-        {"id": pid, "title": f"Title of {pid}"} for pid in paper_ids
-    ]
-    return conn
-
-
-def _sample_graph() -> nx.DiGraph:
-    """A→B, A→C, B→C, B→D, C→D — 5 nodes, D is dangling (not in papers)."""
-    g = nx.DiGraph()
-    g.add_edges_from([("A", "B"), ("A", "C"), ("B", "C"), ("B", "D"), ("C", "D")])
-    return g
-
-
-# ---------------------------------------------------------------------------
-# codex graph report
-# ---------------------------------------------------------------------------
-
-
-class TestGraphReport:
-    def _run(self, graph=None, known_ids=("A", "B", "C"), extra_args=()):
-        if graph is None:
-            graph = _sample_graph()
-        conn = _make_conn_with_paper_ids(*known_ids)
-
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.graph.build_citation_graph", return_value=graph),
-        ):
-            return runner.invoke(app, ["graph", "report", *extra_args])
-
-    def test_exit_zero(self):
-        assert self._run().exit_code == 0
-
-    def test_shows_hub_section(self):
-        assert "HUB" in self._run().output.upper()
-
-    def test_shows_dangling_section(self):
-        assert "DANGLING" in self._run().output.upper()
-
-    def test_dangling_node_in_output(self):
-        # D is cited but not in known_ids
-        assert "D" in self._run().output
-
-    def test_json_output_valid(self):
-        import json
-
-        graph = _sample_graph()
-        conn = _make_conn_with_paper_ids("A", "B", "C")
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.graph.build_citation_graph", return_value=graph),
-        ):
-            result = runner.invoke(app, ["graph", "report", "--json"])
-        assert result.exit_code == 0
-        data = json.loads(result.output)
-        assert "hubs" in data
-        assert "dangling" in data
-        assert "nodes" in data
-        assert data["dangling_count"] == 1
-
-    def test_empty_graph_message(self):
-        conn = _make_conn_with_paper_ids()
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.graph.build_citation_graph", return_value=nx.DiGraph()),
-        ):
-            result = runner.invoke(app, ["graph", "report"])
-        assert result.exit_code == 0
-        assert "empty" in result.output.lower()
-
-    def test_top_n_respected(self):
-        # Build graph with 8 nodes, ask for top 3
-        g = nx.DiGraph()
-        for i in range(8):
-            g.add_edge(f"P{i}", "HUB")
-        conn = _make_conn_with_paper_ids(*[f"P{i}" for i in range(8)], "HUB")
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.graph.build_citation_graph", return_value=g),
-        ):
-            result = runner.invoke(app, ["graph", "report", "--top-n", "3"])
-        assert result.exit_code == 0
-
-    def test_small_corpus_warning_shown(self):
-        # known_ids has only 3 papers (< graph_min_corpus_size default 15)
-        result = self._run(known_ids=("A", "B", "C"))
-        assert result.exit_code == 0
-        assert "Warning" in result.output or "warning" in result.output
-
-    def test_shows_citing_coverage_line(self):
-        # _sample_graph: A, B, C all cite → 3/3 = 100%, so the line shows but no warning.
-        out = self._run(known_ids=("A", "B", "C")).output
-        assert "Citing coverage" in out
-        assert "weakens" not in out  # 100% coverage → no R-D warning
-
-    def test_low_citing_coverage_warning(self):
-        # Only A has an out-edge; B and C are ingested but cite nothing → 1/3 = 33%.
-        g = nx.DiGraph()
-        g.add_edge("A", "X")
-        g.add_nodes_from(["B", "C"])
-        result = self._run(graph=g, known_ids=("A", "B", "C"))
-        assert result.exit_code == 0
-        assert "weakens PageRank" in result.output  # the R-D coverage warning
-
-    def test_json_includes_citing_coverage(self):
-        import json
-
-        g = nx.DiGraph()
-        g.add_edge("A", "X")
-        g.add_nodes_from(["B", "C"])
-        conn = _make_conn_with_paper_ids("A", "B", "C")
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.graph.build_citation_graph", return_value=g),
-        ):
-            result = runner.invoke(app, ["graph", "report", "--json"])
-        data = json.loads(result.output)
-        assert data["citing_coverage"] == {"citing": 1, "papers": 3, "fraction": 0.3333}
-
-
-# ---------------------------------------------------------------------------
-# codex graph related
-# ---------------------------------------------------------------------------
-
-
-class TestGraphRelated:
-    def _run(self, paper_id: str, graph=None, extra_args=()):
-        if graph is None:
-            graph = _sample_graph()
-        conn = MagicMock()
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.graph.build_citation_graph", return_value=graph),
-        ):
-            return runner.invoke(app, ["graph", "related", paper_id, *extra_args])
-
-    def test_exit_zero_known_paper(self):
-        assert self._run("A").exit_code == 0
-
-    def test_unknown_paper_graceful(self):
-        result = self._run("UNKNOWN_ID")
-        assert result.exit_code == 0
-        assert "No papers" in result.output
-
-    def test_no_results_message_when_min_shared_high(self):
-        result = self._run("A", _sample_graph(), extra_args=("--min-shared", "99"))
-        assert "No papers" in result.output
-
-    def test_related_papers_shown(self):
-        # A cites {B,C}; B also cites C → shared=1 with A
-        result = self._run("A", _sample_graph(), extra_args=("--min-shared", "1"))
-        assert result.exit_code == 0
-        # B shares C with A; should appear
-        assert "B" in result.output
-
-
-# ---------------------------------------------------------------------------
-# codex search paper --cite-boost
-# ---------------------------------------------------------------------------
-
-
-class TestSearchCiteBoost:
-    def _make_two_paper_conn(self, pr_winner: str) -> MagicMock:
-        """Two papers A and B at equal distance; pr_winner has a high PageRank score."""
-        conn = MagicMock()
-        conn.execute.return_value.fetchall.return_value = [
-            {"id": "A", "title": "Paper A", "year": 2020, "distance": 0.5},
-            {"id": "B", "title": "Paper B", "year": 2021, "distance": 0.5},
-        ]
-        return conn
-
-    def test_cite_boost_exits_zero(self):
-        conn = MagicMock()
-        conn.execute.return_value.fetchall.return_value = [
-            {"id": "A", "title": "Paper A", "year": 2020, "distance": 0.2},
-        ]
-        graph = _sample_graph()
-
-        def _fake_encode(texts):
-            return np.zeros((len(texts), 1024), dtype="float32")
-
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.embed.get_embedder") as mock_emb,
-            patch("codex.graph.build_citation_graph", return_value=graph),
-        ):
-            mock_emb.return_value.encode_dense.side_effect = _fake_encode
-            result = runner.invoke(app, ["search", "paper", "hodge star", "--cite-boost"])
-        assert result.exit_code == 0
-
-    def test_cite_boost_promotes_high_pr_paper(self):
-        """High-PR paper should appear first (lower boosted distance) via divide formula."""
-        import networkx as nx
-
-        # Build a graph where A has high PageRank (many papers cite A)
-        g = nx.DiGraph()
-        for i in range(10):
-            g.add_edge(f"P{i}", "A")  # A is heavily cited → high PR
-        g.add_edge("P0", "B")  # B is barely cited → low PR
-
-        conn = MagicMock()
-        conn.execute.return_value.fetchall.return_value = [
-            {"id": "A", "title": "Hub", "year": 2020, "distance": 0.5},
-            {"id": "B", "title": "Niche", "year": 2021, "distance": 0.5},
-        ]
-
-        def _fake_encode(texts):
-            return np.zeros((len(texts), 1024), dtype="float32")
-
-        with (
-            patch("codex.db.get_conn", side_effect=_make_conn_cm(conn)),
-            patch("codex.embed.get_embedder") as mock_emb,
-            patch("codex.graph.build_citation_graph", return_value=g),
-        ):
-            mock_emb.return_value.encode_dense.side_effect = _fake_encode
-            result = runner.invoke(app, ["search", "paper", "query", "--cite-boost"])
-
-        assert result.exit_code == 0
-        lines = [ln for ln in result.output.strip().splitlines() if "Hub" in ln or "Niche" in ln]
-        assert len(lines) == 2
-        # A (Hub) should appear before B (Niche) — lower boosted distance
-        hub_line = next(line for line in lines if "Hub" in line)
-        niche_line = next(line for line in lines if "Niche" in line)
-        assert lines.index(hub_line) < lines.index(niche_line)

tests/graph/test_graph.py

@@ -1,311 +0,0 @@
-"""Tests for codex.graph — citation graph analytics (F-15)."""
-
-from __future__ import annotations
-
-from unittest.mock import MagicMock
-
-import networkx as nx
-import pytest
-
-from codex.graph import (
-    build_citation_graph,
-    citation_pagerank,
-    citing_coverage,
-    dangling_citations,
-    find_co_cited,
-    find_related,
-)
-
-# ---------------------------------------------------------------------------
-# Fixtures
-# ---------------------------------------------------------------------------
-
-
-def _make_conn(rows: list[dict]) -> MagicMock:
-    conn = MagicMock()
-    conn.execute.return_value.fetchall.return_value = rows
-    return conn
-
-
-def _citation_rows(*pairs: tuple[str, str]) -> list[dict]:
-    return [{"citing_id": a, "cited_id": b} for a, b in pairs]
-
-
-def _small_graph() -> nx.DiGraph:
-    """
-    A→C, A→D, B→C, B→D, B→E, C→D
-    Nodes: A B C D E  (5 nodes ingested)
-    """
-    rows = _citation_rows(
-        ("A", "C"),
-        ("A", "D"),
-        ("B", "C"),
-        ("B", "D"),
-        ("B", "E"),
-        ("C", "D"),
-    )
-    return build_citation_graph(_make_conn(rows))
-
-
-# ---------------------------------------------------------------------------
-# build_citation_graph
-# ---------------------------------------------------------------------------
-
-
-class TestBuildCitationGraph:
-    def test_empty_db_returns_empty_graph(self):
-        g = build_citation_graph(_make_conn([]))
-        assert isinstance(g, nx.DiGraph)
-        assert g.number_of_nodes() == 0
-        assert g.number_of_edges() == 0
-
-    def test_nodes_and_edges_populated(self):
-        rows = _citation_rows(("P1", "P2"), ("P1", "P3"), ("P2", "P3"))
-        g = build_citation_graph(_make_conn(rows))
-        assert set(g.nodes()) == {"P1", "P2", "P3"}
-        assert g.number_of_edges() == 3
-
-    def test_edge_direction(self):
-        rows = _citation_rows(("citing", "cited"))
-        g = build_citation_graph(_make_conn(rows))
-        assert g.has_edge("citing", "cited")
-        assert not g.has_edge("cited", "citing")
-
-    def test_duplicate_edges_collapsed(self):
-        rows = _citation_rows(("A", "B"), ("A", "B"))
-        g = build_citation_graph(_make_conn(rows))
-        assert g.number_of_edges() == 1
-
-    def test_dangling_nodes_included(self):
-        rows = _citation_rows(("P1", "not_in_papers"))
-        g = build_citation_graph(_make_conn(rows))
-        assert "not_in_papers" in g.nodes()
-
-    def test_cross_citation_uses_doi_when_in_kb(self):
-        # Simulate DB COALESCE: cited_id already resolved to papers.id (DOI)
-        # for in-KB papers; dangling references keep their OpenAlex ID.
-        rows = [
-            {
-                "citing_id": "https://doi.org/10.1234/paper-a",
-                "cited_id": "https://doi.org/10.5678/paper-b",
-            },
-            {
-                "citing_id": "https://doi.org/10.5678/paper-b",
-                "cited_id": "https://openalex.org/W999",
-            },
-        ]
-        g = build_citation_graph(_make_conn(rows))
-        assert g.has_edge("https://doi.org/10.1234/paper-a", "https://doi.org/10.5678/paper-b")
-        assert g.has_edge("https://doi.org/10.5678/paper-b", "https://openalex.org/W999")
-        assert g.number_of_nodes() == 3
-
-
-# ---------------------------------------------------------------------------
-# citation_pagerank
-# ---------------------------------------------------------------------------
-
-
-class TestCitationPagerank:
-    def test_empty_graph_returns_empty_dict(self):
-        g = nx.DiGraph()
-        assert citation_pagerank(g) == {}
-
-    def test_scores_sum_to_one(self):
-        g = _small_graph()
-        pr = citation_pagerank(g)
-        assert abs(sum(pr.values()) - 1.0) < 1e-6
-
-    def test_hub_node_has_highest_score(self):
-        # D is pointed to by A, B, C → should rank highest
-        g = _small_graph()
-        pr = citation_pagerank(g)
-        assert pr["D"] == max(pr.values())
-
-    def test_all_nodes_present(self):
-        g = _small_graph()
-        pr = citation_pagerank(g)
-        assert set(pr.keys()) == set(g.nodes())
-
-    def test_small_graph_returns_uniform_scores(self):
-        # 3 nodes < _MIN_PAGERANK_NODES (5) → uniform
-        rows = _citation_rows(("A", "B"), ("B", "C"))
-        g = build_citation_graph(_make_conn(rows))
-        pr = citation_pagerank(g)
-        assert len(pr) == 3
-        scores = list(pr.values())
-        assert all(abs(s - scores[0]) < 1e-9 for s in scores)
-
-    def test_small_graph_logs_warning(self, caplog: pytest.LogCaptureFixture):
-        import logging
-
-        rows = _citation_rows(("A", "B"), ("B", "C"))
-        g = build_citation_graph(_make_conn(rows))
-        with caplog.at_level(logging.WARNING, logger="codex.graph"):
-            citation_pagerank(g)
-        assert any("corpus too small" in rec.message for rec in caplog.records)
-
-    def test_custom_damping(self):
-        g = _small_graph()
-        pr_85 = citation_pagerank(g, damping=0.85)
-        pr_50 = citation_pagerank(g, damping=0.50)
-        # Different damping → different scores (not all equal)
-        assert pr_85 != pr_50
-
-
-# ---------------------------------------------------------------------------
-# find_related  (bibliographic coupling)
-# ---------------------------------------------------------------------------
-
-
-class TestFindRelated:
-    def _coupling_graph(self) -> nx.DiGraph:
-        """
-        A cites: X Y Z
-        B cites: X Y
-        C cites: X
-        D cites: W   (no overlap with A)
-        """
-        rows = _citation_rows(
-            ("A", "X"),
-            ("A", "Y"),
-            ("A", "Z"),
-            ("B", "X"),
-            ("B", "Y"),
-            ("C", "X"),
-            ("D", "W"),
-        )
-        return build_citation_graph(_make_conn(rows))
-
-    def test_unknown_paper_returns_empty(self):
-        g = self._coupling_graph()
-        assert find_related("UNKNOWN", g) == []
-
-    def test_paper_with_no_refs_returns_empty(self):
-        # X has no outgoing edges → no references to couple on
-        g = self._coupling_graph()
-        assert find_related("X", g) == []
-
-    def test_related_by_two_shared_refs(self):
-        g = self._coupling_graph()
-        related = find_related("A", g, min_shared=2)
-        assert "B" in related
-
-    def test_not_related_below_threshold(self):
-        g = self._coupling_graph()
-        related = find_related("A", g, min_shared=2)
-        assert "C" not in related  # only 1 shared ref
-
-    def test_ordered_by_shared_count_desc(self):
-        g = self._coupling_graph()
-        related = find_related("A", g, min_shared=1)
-        # B shares 2 refs, C shares 1 → B before C
-        assert related.index("B") < related.index("C")
-
-    def test_self_not_in_results(self):
-        g = self._coupling_graph()
-        related = find_related("A", g, min_shared=1)
-        assert "A" not in related
-
-
-# ---------------------------------------------------------------------------
-# find_co_cited
-# ---------------------------------------------------------------------------
-
-
-class TestFindCoCited:
-    def _co_citation_graph(self) -> nx.DiGraph:
-        """
-        P1 cites: A, B, C
-        P2 cites: A, B
-        P3 cites: A
-        """
-        rows = _citation_rows(
-            ("P1", "A"),
-            ("P1", "B"),
-            ("P1", "C"),
-            ("P2", "A"),
-            ("P2", "B"),
-            ("P3", "A"),
-        )
-        return build_citation_graph(_make_conn(rows))
-
-    def test_unknown_paper_returns_empty(self):
-        g = self._co_citation_graph()
-        assert find_co_cited("UNKNOWN", g) == []
-
-    def test_paper_with_no_citers_returns_empty(self):
-        # C has citers but let's test a node that has no predecessors
-        rows = _citation_rows(("P1", "X"))
-        g = build_citation_graph(_make_conn(rows))
-        # "X" is cited once by P1, but P1 has no other citations
-        # X is co-cited with nothing
-        result = find_co_cited("X", g)
-        assert result == []
-
-    def test_most_co_cited_first(self):
-        g = self._co_citation_graph()
-        result = find_co_cited("A", g)
-        # B is co-cited twice (by P1 and P2); C once (by P1 only)
-        ids = [r[0] for r in result]
-        assert ids[0] == "B"
-
-    def test_self_not_included(self):
-        g = self._co_citation_graph()
-        result = find_co_cited("A", g)
-        assert all(pid != "A" for pid, _ in result)
-
-    def test_counts_correct(self):
-        g = self._co_citation_graph()
-        result = dict(find_co_cited("A", g))
-        assert result["B"] == 2  # P1 and P2 both cite A and B
-        assert result["C"] == 1  # only P1 cites A and C
-
-
-# ---------------------------------------------------------------------------
-# dangling_citations
-# ---------------------------------------------------------------------------
-
-
-class TestDanglingCitations:
-    def test_empty_graph(self):
-        assert dangling_citations(nx.DiGraph(), set()) == []
-
-    def test_all_known_returns_empty(self):
-        rows = _citation_rows(("A", "B"), ("B", "C"))
-        g = build_citation_graph(_make_conn(rows))
-        assert dangling_citations(g, {"A", "B", "C"}) == []
-
-    def test_identifies_dangling(self):
-        rows = _citation_rows(("A", "B"), ("A", "external_doi"))
-        g = build_citation_graph(_make_conn(rows))
-        dangling = dangling_citations(g, {"A", "B"})
-        assert "external_doi" in dangling
-        assert "A" not in dangling
-        assert "B" not in dangling
-
-    def test_empty_known_set(self):
-        rows = _citation_rows(("A", "B"))
-        g = build_citation_graph(_make_conn(rows))
-        dangling = dangling_citations(g, set())
-        assert set(dangling) == {"A", "B"}
-
-
-# ---------------------------------------------------------------------------
-# citing_coverage (R-D)
-# ---------------------------------------------------------------------------
-
-
-class TestCitingCoverage:
-    def test_counts_papers_with_out_edges(self):
-        # _small_graph: A→.., B→.., C→D have out-edges; D, E have none.
-        g = _small_graph()
-        n_citing, n_papers = citing_coverage(g, {"A", "B", "C", "D", "E"})
-        assert (n_citing, n_papers) == (3, 5)
-
-    def test_paper_absent_from_graph_counts_as_no_out_edges(self):
-        # A cites; Z was ingested but has no chunks/edges, so it isn't a graph node.
-        g = _small_graph()
-        assert citing_coverage(g, {"A", "Z"}) == (1, 2)
-
-    def test_empty(self):
-        assert citing_coverage(nx.DiGraph(), set()) == (0, 0)

tests/ingest/test_ingest.py

@@ -1,942 +0,0 @@
-"""Tests for codex.ingest — end-to-end idempotent ingest pipeline.
-
-All external calls (DB, OpenAlex, S2, embed, parsing) are mocked so
-the suite runs offline in milliseconds.
-"""
-
-from __future__ import annotations
-
-from contextlib import contextmanager
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-import httpx
-import numpy as np
-import psycopg
-import pytest
-
-from codex.ingest import IngestResult, _make_bibkey, ingest_paper
-from codex.models import Citation, FigureChunk, FormulaChunk, Paper
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _make_conn_cm(mock_conn: MagicMock) -> Any:
-    """Return a context-manager factory that yields *mock_conn*."""
-
-    @contextmanager  # type: ignore[arg-type]
-    def _cm() -> Any:
-        yield mock_conn
-
-    return _cm
-
-
-def _make_paper(
-    paper_id: str = "2301.07041",
-    openalex_id: str | None = "W123",
-    abstract: str | None = "Test abstract.",
-) -> Paper:
-    return Paper(
-        id=paper_id,
-        title="A Test Paper",
-        openalex_id=openalex_id,
-        authors=["Alice", "Bob"],
-        year=2023,
-        abstract=abstract,
-    )
-
-
-def _fake_embedder(dim: int = 1024) -> MagicMock:
-    """Return a mock Embedder whose encode_dense returns deterministic output."""
-    embedder = MagicMock()
-    embedder.dim = dim
-
-    def encode_dense(texts: list[str]) -> np.ndarray:
-        return np.ones((len(texts), dim), dtype=np.float32)
-
-    embedder.encode_dense.side_effect = encode_dense
-    return embedder
-
-
-# ---------------------------------------------------------------------------
-# 1. test_ingest_paper_basic
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_basic() -> None:
-    """fetch_paper returns a Paper; DB upsert is called; IngestResult has correct counts."""
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    # OpenAlex returns citing_id=openalex_id; ingest.py must rewrite to paper.id (D-05c).
-    raw_api_citations = [
-        Citation(citing_id=paper.openalex_id or "", cited_id="W999"),
-    ]
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper) as mock_fetch,
-        patch("codex.ingest.openalex.fetch_citations", return_value=raw_api_citations),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result = ingest_paper(paper.id)
-
-    mock_fetch.assert_called_once_with(paper.id)
-
-    # Paper upsert must have been issued
-    assert mock_conn.execute.call_count >= 1
-    first_sql: str = mock_conn.execute.call_args_list[0][0][0]
-    assert "INSERT INTO papers" in first_sql
-    assert "ON CONFLICT" in first_sql
-
-    assert isinstance(result, IngestResult)
-    assert result.paper_id == paper.id
-    assert result.chunks_upserted == 0
-    assert result.citations_upserted == len(raw_api_citations)
-
-    # D-05c regression: citing_id must be rewritten to paper.id (DOI), not openalex_id.
-    mock_cursor = mock_conn.cursor.return_value.__enter__.return_value
-    cit_call = mock_cursor.executemany.call_args_list[0]
-    inserted_rows: list[tuple[str, str, Any]] = cit_call[0][1]
-    assert inserted_rows[0][0] == paper.id, (
-        f"citing_id must be paper.id='{paper.id}', not openalex_id='{paper.openalex_id}'"
-    )
-
-
-def test_ingest_citation_fallback_to_crossref() -> None:
-    """R-B: OpenAlex + S2 both empty → Crossref references supply the edges.
-
-    Crossref is the third leg of the fallback chain. Its cited DOIs are
-    case-normalised and citing_id is rewritten to the canonical paper.id.
-    """
-    paper = _make_paper(paper_id="10.1007/s00454-019-00132-8", openalex_id="W123")
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    # Upper-case suffix proves the cited DOI is normalised to bare lower-case.
-    crossref_refs = [Citation(citing_id=paper.id, cited_id="10.1090/S0002-9947-04-03545-7")]
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),  # OpenAlex empty
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=[]),  # S2 empty
-        patch("codex.ingest.crossref.fetch_references", return_value=crossref_refs) as mock_cr,
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result = ingest_paper(paper.id)
-
-    # Crossref consulted with the bare DOI; one edge upserted.
-    mock_cr.assert_called_once_with(paper.id)
-    assert result.citations_upserted == 1
-    mock_cursor = mock_conn.cursor.return_value.__enter__.return_value
-    cit_rows = mock_cursor.executemany.call_args_list[0][0][1]
-    assert cit_rows[0][0] == paper.id
-    assert cit_rows[0][1] == "10.1090/s0002-9947-04-03545-7"
-
-
-def test_ingest_crossref_not_called_when_openalex_has_citations() -> None:
-    """R-B: the Crossref leg fires only when OpenAlex+S2 are empty (it's a fallback)."""
-    paper = _make_paper(paper_id="10.1007/s00454-019-00132-8", openalex_id="W123")
-    mock_conn = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch(
-            "codex.ingest.openalex.fetch_citations",
-            return_value=[Citation(citing_id="W123", cited_id="https://openalex.org/W9")],
-        ),
-        patch("codex.ingest.crossref.fetch_references") as mock_cr,
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        ingest_paper(paper.id)
-
-    mock_cr.assert_not_called()
-
-
-def test_ingest_canonicalises_paper_id_to_caller_arg() -> None:
-    """papers.id is the caller-supplied id, not OpenAlex's URL-form id (audit C-7).
-
-    Real OpenAlex returns the doi/id as a full URL; ingest must key the paper on
-    the bare id the caller passed (matching ingest scripts and CLI lookups), and
-    keep the OpenAlex work id only as openalex_id.
-    """
-    oa_paper = Paper(
-        id="https://doi.org/10.48550/arxiv.math/0603097",  # URL form, as OpenAlex emits
-        title="A Test Paper",
-        openalex_id="https://openalex.org/W4301005794",
-        authors=["Alice", "Bob"],
-        year=2008,
-        abstract="Test abstract.",
-    )
-    mock_conn = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=oa_paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result = ingest_paper("math/0603097")
-
-    papers_params = mock_conn.execute.call_args_list[0][0][1]
-    assert papers_params["id"] == "math/0603097", "papers.id must be the caller's bare id"
-    assert papers_params["openalex_id"] == "https://openalex.org/W4301005794"
-    assert result.paper_id == "math/0603097"
-
-
-# ---------------------------------------------------------------------------
-# 2. test_ingest_paper_source_tex
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_source_tex(tmp_path: Any) -> None:
-    """``.tex`` source → section-aware chunking; the stored ``section`` column is
-    labelled from the real ``\\section`` heading, not the chunk body (R-C / R-12)."""
-    tex_file = tmp_path / "paper.tex"
-    tex_file.write_text(r"\section{Introduction} Hello world. This is a test paper.")
-
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    # (section title, chunk content) pairs from the section-aware chunker.
-    section_pairs = [
-        ("Introduction", "Body of the introduction goes here."),
-        ("Proof of the Main Theorem", "Body of the proof goes here."),
-        ("3.2 Discrete Conformal Maps", "Body about discrete conformal maps."),
-    ]
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-        patch("codex.parsing.tex.chunk_sections", return_value=section_pairs),
-        patch("codex.ingest.is_quality_chunk", return_value=True),
-    ):
-        result = ingest_paper(paper.id, source_path=str(tex_file))
-
-    # DELETE + INSERT for chunks
-    sql_calls = [str(c[0][0]) for c in mock_conn.execute.call_args_list]
-    assert any("DELETE FROM chunks" in sql for sql in sql_calls)
-
-    # executemany is called on the cursor, not on the connection directly
-    mock_cursor = mock_conn.cursor.return_value.__enter__.return_value
-    assert mock_cursor.executemany.call_count >= 1
-    chunk_insert_call = mock_cursor.executemany.call_args_list[0]
-    chunk_sql: str = chunk_insert_call[0][0]
-    assert "INSERT INTO chunks" in chunk_sql
-
-    # R-F: a bare canonical heading → bucket ("Introduction"→intro); any other
-    # heading is stored as its cleaned real title ("proof of the main theorem",
-    # "discrete conformal maps") instead of being collapsed/mislabelled.
-    chunk_rows = chunk_insert_call[0][1]
-    assert [row[4] for row in chunk_rows] == [
-        "intro",
-        "proof of the main theorem",
-        "discrete conformal maps",
-    ]
-    assert result.chunks_upserted == len(section_pairs)
-
-
-# ---------------------------------------------------------------------------
-# 3. test_ingest_paper_source_pdf
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_source_pdf(tmp_path: Any) -> None:
-    """``.pdf`` source → pdf_to_markdown, chunk_text, embed, grobid refs, chunks + citations."""
-    pdf_file = tmp_path / "paper.pdf"
-    pdf_file.write_bytes(b"%PDF-1.4 fake content")
-
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    fake_chunks = ["pdf chunk one.", "pdf chunk two."]
-    grobid_refs = [
-        {"title": "Ref A", "authors": "X", "year": "2020", "doi": "10.1/ref_a", "arxiv_id": ""},
-        {"title": "Ref B", "authors": "Y", "year": "2021", "doi": "", "arxiv_id": "2101.00001"},
-    ]
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        # OpenAlex empty → ingest now consults the S2 reference supplement (DQ-1);
-        # mock it empty so only the GROBID refs contribute to the count.
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-        patch("codex.parsing.nougat.pdf_to_markdown", return_value="pdf markdown text."),
-        patch("codex.parsing.tex.chunk_text", return_value=fake_chunks),
-        patch("codex.parsing.grobid.extract_references", return_value=grobid_refs),
-        patch("codex.ingest.is_quality_chunk", return_value=True),
-    ):
-        result = ingest_paper(paper.id, source_path=str(pdf_file))
-
-    # Chunks were inserted
-    assert result.chunks_upserted == len(fake_chunks)
-
-    # GROBID refs were merged into citations
-    # Two grobid refs → both have either doi or arxiv_id → 2 citations
-    assert result.citations_upserted == 2
-
-    # executemany is called on the cursor for chunks + citations (at least 2 calls total)
-    mock_cursor = mock_conn.cursor.return_value.__enter__.return_value
-    assert mock_cursor.executemany.call_count >= 2
-
-
-def test_ingest_pdf_grobid_citations_normalize_doi(tmp_path: Any) -> None:
-    """R-A guard: GROBID extracts the DOI verbatim from the PDF reference text, so
-    it may be mixed-case, a ``https://doi.org/…`` URL, or ``doi:``-prefixed. Each
-    must be normalized to the bare, lower-cased canonical form before the Citation
-    is built — otherwise the same reference splits into case-/URL-variant graph
-    nodes that never join the bare-lowercase ``papers.id``. arXiv-only refs (no
-    DOI) are left untouched. Mirrors the S2-supplement normalization (DQ-1)."""
-    pdf_file = tmp_path / "paper.pdf"
-    pdf_file.write_bytes(b"%PDF-1.4 fake content")
-
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    def _ref(doi: str = "", arxiv_id: str = "") -> dict[str, str]:
-        # Faithful to GROBID's output shape: all five keys always present.
-        return {"title": "", "authors": "", "year": "", "doi": doi, "arxiv_id": arxiv_id}
-
-    grobid_refs = [
-        _ref(doi="10.1007/S00454-019-00132-8"),  # mixed-case bare DOI
-        _ref(doi="https://doi.org/10.1145/AbC.123"),  # URL-form DOI
-        _ref(doi="doi:10.1/MixedCase"),  # doi:-prefixed DOI
-        _ref(arxiv_id="2101.00001"),  # arXiv-only ref → untouched
-    ]
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        # OpenAlex empty → S2 supplement consulted (DQ-1); empty so only GROBID
-        # refs contribute to the citation rows under inspection.
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-        patch("codex.parsing.nougat.pdf_to_markdown", return_value="pdf markdown text."),
-        patch("codex.parsing.tex.chunk_text", return_value=["pdf chunk one."]),
-        patch("codex.parsing.grobid.extract_references", return_value=grobid_refs),
-        patch("codex.ingest.is_quality_chunk", return_value=True),
-    ):
-        result = ingest_paper(paper.id, source_path=str(pdf_file))
-
-    assert result.citations_upserted == 4
-
-    # Locate the citations INSERT among the cursor's executemany calls (chunks
-    # are also inserted via executemany, so filter by SQL rather than by index).
-    mock_cursor = mock_conn.cursor.return_value.__enter__.return_value
-    citation_calls = [
-        c for c in mock_cursor.executemany.call_args_list if "INSERT INTO citations" in c[0][0]
-    ]
-    assert len(citation_calls) == 1
-    inserted_rows: list[tuple[str, str, Any]] = citation_calls[0][0][1]
-
-    # Every edge hangs off the canonical paper.id (citing side).
-    assert all(row[0] == paper.id for row in inserted_rows)
-
-    cited_ids = {row[1] for row in inserted_rows}
-    assert cited_ids == {
-        "10.1007/s00454-019-00132-8",  # mixed-case → lower-cased
-        "10.1145/abc.123",  # URL prefix stripped + lower-cased
-        "10.1/mixedcase",  # doi: prefix stripped + lower-cased
-        "2101.00001",  # arXiv id untouched
-    }
-    # No raw URL-form / prefixed DOI leaked into the citation graph.
-    assert not any(cid.startswith(("http", "doi:")) for cid in cited_ids)
-
-
-# ---------------------------------------------------------------------------
-# 4. test_ingest_paper_not_found
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_not_found() -> None:
-    """OpenAlex returns None for unknown ID → ValueError raised."""
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=None),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        pytest.raises(ValueError, match="Paper not found"),
-    ):
-        ingest_paper("10.9999/unknown-doi")
-
-
-# ---------------------------------------------------------------------------
-# 5. test_ingest_paper_idempotent
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_idempotent() -> None:
-    """Second call with the same paper_id overwrites without error (no unique violations)."""
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result1 = ingest_paper(paper.id)
-        result2 = ingest_paper(paper.id)
-
-    assert result1.paper_id == result2.paper_id == paper.id
-
-    # ON CONFLICT … DO UPDATE means no error on second call
-    all_execute_calls = mock_conn.execute.call_args_list
-    paper_upserts = [c for c in all_execute_calls if "INSERT INTO papers" in str(c[0][0])]
-    # Two calls → two paper upserts
-    assert len(paper_upserts) == 2
-
-
-# ---------------------------------------------------------------------------
-# 6. test_ingest_paper_arxiv_s2_fallback
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_arxiv_s2_fallback() -> None:
-    """OpenAlex None + arXiv has nothing → stub Paper; S2 references still consulted."""
-    paper_id = "2301.07041"
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=None),
-        # arXiv + abstract sources also empty → genuine stub fallback path.
-        patch("codex.ingest.arxiv.fetch_metadata", return_value=None) as mock_arxiv,
-        patch("codex.ingest.semanticscholar.fetch_abstract", return_value=None),
-        patch("codex.ingest.crossref.fetch_abstract", return_value=None),
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=[]) as mock_s2,
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result = ingest_paper(paper_id)
-
-    # arXiv metadata recovery attempted; S2 references consulted as fallback.
-    mock_arxiv.assert_called_once_with(paper_id)
-    mock_s2.assert_called()
-    assert result.paper_id == paper_id
-
-
-def test_ingest_paper_openalex_404_recovers_from_arxiv() -> None:
-    """DQ-2: OpenAlex 404 on an arXiv id → authoritative metadata from the arXiv API,
-    bibkey auto-generated, abstract embedded (no zero vector)."""
-    recovered = Paper(
-        id="1911.00966",
-        title="A discrete version of Liouville's theorem on conformal maps",
-        authors=["Ulrich Pinkall", "Boris Springborn"],
-        year=2019,
-        abstract="Liouville's theorem says that in dimension greater than two...",
-    )
-    fake_emb = _fake_embedder()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=None),
-        patch("codex.ingest.arxiv.fetch_metadata", return_value=recovered) as mock_arxiv,
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=fake_emb),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result = ingest_paper("1911.00966")
-
-    mock_arxiv.assert_called_once_with("1911.00966")
-    assert result.paper_id == "1911.00966"
-    params: dict[str, Any] = mock_conn.execute.call_args_list[0][0][1]
-    assert params["title"].startswith("A discrete version")
-    assert params["bibkey"] == "PinkallSpringborn2019"  # generated from recovered authors+year
-    # abstract present → real embedding computed, not a zero vector
-    fake_emb.encode_dense.assert_called_once()
-
-
-def test_ingest_paper_empty_abstract_recovered_from_s2() -> None:
-    """DQ-2: OpenAlex has the paper but no abstract → supplemented from S2 before embedding."""
-    paper = _make_paper(abstract=None)  # has openalex_id, no abstract
-    fake_emb = _fake_embedder()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=[]),
-        patch(
-            "codex.ingest.semanticscholar.fetch_abstract",
-            return_value="Recovered abstract text.",
-        ) as mock_abs,
-        patch("codex.ingest.get_embedder", return_value=fake_emb),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        ingest_paper(paper.id)
-
-    mock_abs.assert_called_once()
-    params: dict[str, Any] = mock_conn.execute.call_args_list[0][0][1]
-    assert params["abstract"] == "Recovered abstract text."
-    fake_emb.encode_dense.assert_called_once_with(["Recovered abstract text."])
-
-
-def test_ingest_paper_openalex_empty_uses_s2_supplement() -> None:
-    """DQ-1: OpenAlex returns an empty reference list → ingest supplements from S2,
-    rewriting citing_id to paper.id and lowercasing DOI cited-ids."""
-    paper = _make_paper()  # openalex_id="W123", id="2301.07041"
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    # S2 keys citing_id by the id form passed in; ingest must canonicalise it.
-    s2_refs = [
-        Citation(citing_id="arXiv:2301.07041", cited_id="10.1/UPPER", context="ctx"),
-        Citation(citing_id="arXiv:2301.07041", cited_id="2202.00002"),
-    ]
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=s2_refs) as mock_s2,
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result = ingest_paper(paper.id)
-
-    # S2 consulted with a namespaced id (a bare arXiv id 404s on S2).
-    mock_s2.assert_called_once_with("arXiv:2301.07041")
-    assert result.citations_upserted == 2
-
-    mock_cursor = mock_conn.cursor.return_value.__enter__.return_value
-    inserted_rows: list[tuple[str, str, Any]] = mock_cursor.executemany.call_args_list[0][0][1]
-    # citing_id rewritten to paper.id, not the S2 id form.
-    assert all(row[0] == paper.id for row in inserted_rows)
-    cited = {row[1] for row in inserted_rows}
-    assert "10.1/upper" in cited  # DOI lowercased
-    assert "2202.00002" in cited  # arXiv id untouched
-
-
-# ---------------------------------------------------------------------------
-# 7. test_ingest_paper_no_abstract_uses_zero_vector
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_no_abstract_uses_zero_vector() -> None:
-    """A paper without an abstract gets a zero embedding vector (no model call)."""
-    paper = _make_paper(abstract=None)
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    fake_emb = _fake_embedder(dim=1024)
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        # No abstract recoverable from any source (DQ-2) → genuine zero-vector case.
-        patch("codex.ingest.semanticscholar.fetch_abstract", return_value=None),
-        patch("codex.ingest.crossref.fetch_abstract", return_value=None),
-        patch("codex.ingest.get_embedder", return_value=fake_emb),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        ingest_paper(paper.id)
-
-    # encode_dense must NOT have been called (no abstract → zero vector shortcut)
-    fake_emb.encode_dense.assert_not_called()
-
-    # Verify zero vector was passed in the paper upsert
-    upsert_call = mock_conn.execute.call_args_list[0]
-    params: dict[str, Any] = upsert_call[0][1]
-    emb: list[float] = params["abstract_emb"]
-    assert len(emb) == 1024
-    assert all(v == 0.0 for v in emb)
-
-
-# ---------------------------------------------------------------------------
-# 8. F-09 Rich Parsing tests
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_paper_rich_calls_formula_and_figure_parsers(tmp_path: Any) -> None:
-    """When rich=True and source is PDF, formula + figure parsers are invoked."""
-    import codex.parsing.figures as _figures_mod
-    import codex.parsing.mathpix as _mathpix_mod
-
-    pdf_file = tmp_path / "paper.pdf"
-    pdf_file.write_bytes(b"%PDF-1.4 fake content")
-
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    # Extractors derive paper_id from the PDF filename stem ("paper"), which differs
-    # from papers.id — ingest must insert papers.id, not this stem (audit C-10).
-    fake_formula = FormulaChunk(paper_id="paper", page=1, raw_latex=r"\alpha", context="test")
-    fake_figure = FigureChunk(
-        paper_id="paper", page=1, image_path="/tmp/fig.png", caption="Figure 1."
-    )
-
-    mock_settings = MagicMock()
-    mock_settings.figures_dir = str(tmp_path / "figures")
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-        patch("codex.parsing.nougat.pdf_to_markdown", return_value=""),
-        patch("codex.parsing.tex.chunk_text", return_value=[]),
-        patch("codex.parsing.grobid.extract_references", return_value=[]),
-        patch.object(
-            _mathpix_mod, "extract_formulas", return_value=[fake_formula]
-        ) as mock_formulas,
-        patch.object(_figures_mod, "extract_figures", return_value=[fake_figure]) as mock_figures,
-        patch("codex.ingest.get_settings", return_value=mock_settings),
-    ):
-        result = ingest_paper(paper.id, source_path=str(pdf_file), rich=True)
-
-    mock_formulas.assert_called_once()
-    mock_figures.assert_called_once()
-    assert result.formulas_upserted == 1
-    assert result.figures_upserted == 1
-
-    # C-10: formula/figure rows must be keyed on papers.id, not the extractor's
-    # filename-stem paper_id, or the FK to papers(id) is violated.
-    cursor = mock_conn.cursor.return_value.__enter__.return_value
-    inserted_paper_ids = {
-        row[0] for call in cursor.executemany.call_args_list for row in call[0][1]
-    }
-    assert inserted_paper_ids == {paper.id}, (
-        f"formula/figure inserts must use papers.id, got {inserted_paper_ids}"
-    )
-
-
-def test_ingest_paper_no_rich_skips_parsers(tmp_path: Any) -> None:
-    """When rich=False (default), formula and figure parsers are NOT called."""
-    import codex.parsing.figures as _figures_mod
-    import codex.parsing.mathpix as _mathpix_mod
-
-    pdf_file = tmp_path / "paper.pdf"
-    pdf_file.write_bytes(b"%PDF-1.4 fake content")
-
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-        patch("codex.parsing.nougat.pdf_to_markdown", return_value=""),
-        patch("codex.parsing.tex.chunk_text", return_value=[]),
-        patch("codex.parsing.grobid.extract_references", return_value=[]),
-        patch.object(_mathpix_mod, "extract_formulas", return_value=[]) as mock_formulas,
-        patch.object(_figures_mod, "extract_figures", return_value=[]) as mock_figures,
-    ):
-        result = ingest_paper(paper.id, source_path=str(pdf_file), rich=False)
-
-    mock_formulas.assert_not_called()
-    mock_figures.assert_not_called()
-    assert result.formulas_upserted == 0
-    assert result.figures_upserted == 0
-
-
-def test_ingest_paper_rich_requires_pdf_source(tmp_path: Any) -> None:
-    """When rich=True but source is a .tex file, formula/figure parsers are NOT called."""
-    import codex.parsing.figures as _figures_mod
-    import codex.parsing.mathpix as _mathpix_mod
-
-    tex_file = tmp_path / "paper.tex"
-    tex_file.write_text(r"\section{Intro}")
-
-    paper = _make_paper()
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-        patch("codex.parsing.tex.latex_to_text", return_value=""),
-        patch("codex.parsing.tex.chunk_text", return_value=[]),
-        patch.object(_mathpix_mod, "extract_formulas", return_value=[]) as mock_formulas,
-        patch.object(_figures_mod, "extract_figures", return_value=[]) as mock_figures,
-    ):
-        result = ingest_paper(paper.id, source_path=str(tex_file), rich=True)
-
-    mock_formulas.assert_not_called()
-    mock_figures.assert_not_called()
-    assert result.formulas_upserted == 0
-    assert result.figures_upserted == 0
-
-
-# ---------------------------------------------------------------------------
-# D-04 — bibkey auto-generation heuristic
-# ---------------------------------------------------------------------------
-
-
-def test_make_bibkey_single_author() -> None:
-    paper = Paper(id="doi:10.1/x", title="T", authors=["John Smith"], year=2023)
-    assert _make_bibkey(paper) == "Smith2023"
-
-
-def test_make_bibkey_two_authors() -> None:
-    paper = Paper(id="doi:10.1/x", title="T", authors=["Alice Foo", "Bob Bar"], year=2015)
-    assert _make_bibkey(paper) == "FooBar2015"
-
-
-def test_make_bibkey_three_authors() -> None:
-    paper = Paper(
-        id="doi:10.1/x",
-        title="T",
-        authors=["Alexander Bobenko", "Ulrich Pinkall", "Boris Springborn"],
-        year=2015,
-    )
-    assert _make_bibkey(paper) == "BobenkoPinkallSpringborn2015"
-
-
-def test_make_bibkey_many_authors_uses_etal() -> None:
-    paper = Paper(
-        id="doi:10.1/x",
-        title="T",
-        authors=["A One", "B Two", "C Three", "D Four"],
-        year=2020,
-    )
-    assert _make_bibkey(paper) == "OneEtAl2020"
-
-
-def test_make_bibkey_no_authors_returns_none() -> None:
-    paper = Paper(id="doi:10.1/x", title="T", authors=[], year=2023)
-    assert _make_bibkey(paper) is None
-
-
-def test_make_bibkey_no_year_returns_none() -> None:
-    paper = Paper(id="doi:10.1/x", title="T", authors=["John Smith"], year=None)
-    assert _make_bibkey(paper) is None
-
-
-def test_ingest_sets_bibkey_when_none(tmp_path: Any) -> None:
-    """ingest_paper auto-generates bibkey when paper has none (D-04)."""
-    paper = _make_paper()  # bibkey=None (default from _make_paper)
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        ingest_paper(paper.id)
-
-    upsert_params: dict[str, Any] = mock_conn.execute.call_args_list[0][0][1]
-    assert upsert_params["bibkey"] is not None, "bibkey must be auto-generated when paper has none"
-    # authors=["Alice", "Bob"], year=2023 → 2 authors ≤ 3 → concat last tokens
-    assert upsert_params["bibkey"] == "AliceBob2023"
-
-
-def test_ingest_doi_paper_upserts_bare_canonical_id() -> None:
-    """DQ-5 regression: re-ingesting an existing DOI paper must be idempotent.
-
-    OpenAlex returns the ``doi`` as a full URL (and possibly mixed-case), but the
-    stored canonical ``papers.id`` is the BARE, lower-cased DOI (M-1 migration).
-    The papers upsert must therefore carry the BARE id into ``%(id)s`` so
-    ``ON CONFLICT (id)`` matches the existing row and UPDATEs it — rather than
-    attempting a fresh INSERT that trips the separate ``papers_openalex_id_key``
-    unique constraint (the UniqueViolation crash).
-
-    The real ``_map_paper``/``fetch_paper`` run here (only the HTTP ``_get``, the
-    embedder and the DB are mocked), so this exercises the actual normalization
-    through the ingest path. The DB is a MagicMock, so it cannot raise the real
-    constraint error; the live re-ingest of 10.1007/s00454-019-00132-8 is the
-    end-to-end UniqueViolation proof (see DATA-QUALITY-2026-06-15.md, DQ-5).
-    """
-    bare = "10.1007/s00454-019-00132-8"
-    work = {
-        "id": "https://openalex.org/W2899262408",
-        # URL form + an upper-case suffix char — the worst case for ON CONFLICT.
-        "doi": "https://doi.org/10.1007/S00454-019-00132-8",
-        "title": "A Test Paper",
-        "publication_year": 2019,
-        "authorships": [{"author": {"display_name": "Alice Smith"}}],
-        "abstract_inverted_index": {"Hello": [0], "world": [1]},
-        # Non-empty so fetch_citations yields edges and the S2 fallback is skipped.
-        "referenced_works": ["https://openalex.org/W1"],
-    }
-
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch(
-            "codex.ingest.openalex._get",
-            return_value=httpx.Response(200, json=work),
-        ),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        result = ingest_paper(bare)
-
-    # The IngestResult and the upsert must use the bare canonical id, not the URL.
-    assert result.paper_id == bare
-
-    upsert_sql: str = mock_conn.execute.call_args_list[0][0][0]
-    upsert_params: dict[str, Any] = mock_conn.execute.call_args_list[0][0][1]
-    assert "INSERT INTO papers" in upsert_sql
-    assert "ON CONFLICT (id)" in upsert_sql
-    assert upsert_params["id"] == bare, (
-        f"upsert id must be the bare canonical DOI '{bare}', "
-        f"not the OpenAlex URL form '{upsert_params['id']}'"
-    )
-
-
-def test_ingest_normalizes_non_bare_caller_id() -> None:
-    """Review #1: a non-bare CALLER id (URL-form / mixed-case DOI) is normalized to the
-    bare canonical form before being pinned to papers.id (the C-7 pin runs it through
-    _norm_cited_id), so a sloppy caller cannot store a URL-form papers.id that would
-    defeat ON CONFLICT (id) idempotency or the ``paper.id.startswith('10.')`` gates."""
-    paper = _make_paper(paper_id="10.1007/s00454-019-00132-8", openalex_id="W1")
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.executemany = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.semanticscholar.fetch_references", return_value=[]),
-        patch("codex.ingest.crossref.fetch_references", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        # Caller passes the URL form with an upper-case suffix char (worst case).
-        result = ingest_paper("https://doi.org/10.1007/S00454-019-00132-8")
-
-    bare = "10.1007/s00454-019-00132-8"
-    assert result.paper_id == bare
-    upsert_params: dict[str, Any] = mock_conn.execute.call_args_list[0][0][1]
-    assert upsert_params["id"] == bare, (
-        f"non-bare caller must be normalized to '{bare}', got '{upsert_params['id']}'"
-    )
-
-
-def test_ingest_disambiguates_bibkey_on_unique_violation() -> None:
-    """A papers.bibkey UNIQUE collision is retried with an a/b/c suffix (audit C-15)."""
-    paper = _make_paper()  # bibkey auto-generates to "AliceBob2023"
-    mock_conn = MagicMock()
-    state = {"first": True}
-
-    def execute_side_effect(sql: str, params: Any = None) -> MagicMock:
-        # The first papers INSERT collides on the bibkey; the retry must succeed.
-        if "INSERT INTO papers" in sql and state["first"]:
-            state["first"] = False
-            raise psycopg.errors.UniqueViolation(
-                'duplicate key value violates unique constraint "papers_bibkey_key"'
-            )
-        return MagicMock()
-
-    mock_conn.execute.side_effect = execute_side_effect
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        ingest_paper(paper.id)
-
-    papers_inserts = [
-        c for c in mock_conn.execute.call_args_list if "INSERT INTO papers" in c[0][0]
-    ]
-    assert len(papers_inserts) == 2, "expected one retry after the bibkey collision"
-    assert papers_inserts[0][0][1]["bibkey"] == "AliceBob2023"
-    assert papers_inserts[1][0][1]["bibkey"] == "AliceBob2023a"
-    mock_conn.rollback.assert_called_once()
-
-
-def test_ingest_openalex_collision_raises_clear_error() -> None:
-    """An openalex_id UNIQUE collision raises a clear, actionable error (audit C-15).
-
-    Unlike a bibkey clash (auto-suffixed), a shared openalex_id means two papers
-    claim the same OpenAlex work — a real data conflict that must surface, not the
-    raw psycopg error.
-    """
-    paper = _make_paper()
-    mock_conn = MagicMock()
-
-    def execute_side_effect(sql: str, params: Any = None) -> MagicMock:
-        if "INSERT INTO papers" in sql:
-            raise psycopg.errors.UniqueViolation(
-                'duplicate key value violates unique constraint "papers_openalex_id_key"'
-            )
-        return MagicMock()
-
-    mock_conn.execute.side_effect = execute_side_effect
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=[]),
-        patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
-        patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
-        pytest.raises(ValueError, match="openalex_id"),
-    ):
-        ingest_paper(paper.id)
-
-
-# ---------------------------------------------------------------------------
-
-
-def test_ingest_result_has_formula_figure_counts() -> None:
-    """IngestResult has formulas_upserted and figures_upserted fields with default 0."""
-    result = IngestResult(paper_id="test", chunks_upserted=0, citations_upserted=0)
-    assert result.formulas_upserted == 0
-    assert result.figures_upserted == 0
-
-    result2 = IngestResult(
-        paper_id="test",
-        chunks_upserted=5,
-        citations_upserted=3,
-        formulas_upserted=10,
-        figures_upserted=2,
-    )
-    assert result2.formulas_upserted == 10
-    assert result2.figures_upserted == 2

tests/integration/test_pipeline_smoke.py

@@ -1,100 +0,0 @@
-"""End-to-end smoke test: ingest → discover → provenance flow (all mocked)."""
-
-from __future__ import annotations
-
-from contextlib import contextmanager
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-import numpy as np
-
-from codex.discover import discovery_leads
-from codex.ingest import ingest_paper
-from codex.models import Citation, Paper
-from codex.provenance import export_bib
-
-
-def _make_conn_cm(mock_conn: MagicMock) -> Any:
-    """Return a context-manager factory that yields *mock_conn*."""
-
-    @contextmanager
-    def _cm() -> Any:
-        yield mock_conn
-
-    return _cm
-
-
-def test_full_pipeline_smoke() -> None:
-    """ingest_paper → discovery_leads → export_bib all execute without errors (all mocked)."""
-    paper = Paper(id="2301.07041", title="T", openalex_id="W123")
-    citations = [Citation(citing_id="2301.07041", cited_id="2301.99999")]
-
-    # --- mock connection setup ---
-    mock_conn = MagicMock()
-    mock_conn.execute = MagicMock()
-    mock_conn.commit = MagicMock()
-
-    # discovery_leads: conn.execute(...).fetchall() → empty list (no leads)
-    discover_execute_result = MagicMock()
-    discover_execute_result.fetchall.return_value = []
-
-    # export_bib: conn.execute(...).fetchall() → one paper row with bibkey
-    bib_row: dict[str, Any] = {
-        "id": "2301.07041",
-        "bibkey": "paper2023",
-        "title": "T",
-        "authors": ["Author One"],
-        "year": 2023,
-    }
-    bib_execute_result = MagicMock()
-    bib_execute_result.fetchall.return_value = [bib_row]
-
-    # The ingest path calls conn.execute multiple times (paper upsert, etc.).
-    # discovery_leads and export_bib each call conn.execute once and use fetchall.
-    # We use side_effect to route calls: first N calls from ingest return a generic
-    # mock; then we supply discover and bib results in order.
-    _call_count: list[int] = [0]
-
-    def _execute_side_effect(*args: Any, **kwargs: Any) -> MagicMock:
-        _call_count[0] += 1
-        sql: str = str(args[0]) if args else ""
-        if "cited_id NOT IN" in sql:
-            return discover_execute_result
-        if "FROM papers" in sql and "bibkey IS NOT NULL" in sql:
-            return bib_execute_result
-        return MagicMock()
-
-    mock_conn.execute.side_effect = _execute_side_effect
-
-    # cursor() context manager for executemany calls inside ingest
-    mock_cursor = MagicMock()
-    mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
-    mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=False)
-
-    embedder = MagicMock()
-    embedder.dim = 1024
-    embedder.encode_dense.return_value = np.zeros((1, 1024), dtype=np.float32)
-
-    conn_cm = _make_conn_cm(mock_conn)
-
-    with (
-        patch("codex.ingest.openalex.fetch_paper", return_value=paper),
-        patch("codex.ingest.openalex.fetch_citations", return_value=citations),
-        patch("codex.ingest.get_embedder", return_value=embedder),
-        patch("codex.ingest.get_conn", side_effect=conn_cm),
-        patch("codex.discover.get_conn", side_effect=conn_cm),
-        patch("codex.provenance.get_conn", side_effect=conn_cm),
-    ):
-        # Step 1: ingest
-        result = ingest_paper("2301.07041")
-
-        # Step 2: discover
-        leads = discovery_leads()
-
-        # Step 3: export bib
-        bib = export_bib(["2301.07041"])
-
-    # --- assertions ---
-    assert result.paper_id == "2301.07041"
-    assert isinstance(leads, list)
-    assert "@article" in bib

tests/mcp/test_graceful.py

@@ -1,211 +0,0 @@
-"""test_graceful.py — missing/unavailable features must degrade, never crash.
-
-All tools that wrap optional features (wiki, synthesis leads, provenance)
-must return a structured "feature not available" response instead of raising.
-"""
-
-from __future__ import annotations
-
-import json
-import tempfile
-from pathlib import Path
-from unittest.mock import patch
-
-# ---------------------------------------------------------------------------
-# ask — always returns "not available" (RAG not implemented)
-# ---------------------------------------------------------------------------
-
-
-def test_ask_returns_not_available() -> None:
-    """ask() must return {'error': 'not available'} — RAG not yet implemented."""
-    from codex.mcp_server import ask
-
-    result = ask("What is conformal geometry?")
-    assert result == {"error": "not available"}
-
-
-# ---------------------------------------------------------------------------
-# wiki_read — page missing → "feature not available"
-# ---------------------------------------------------------------------------
-
-
-def test_wiki_read_missing_page_returns_not_available() -> None:
-    """wiki_read() with a nonexistent page must return {"error": "feature not available"}."""
-    from codex.mcp_server import wiki_read
-
-    with (
-        tempfile.TemporaryDirectory() as tmpdir,
-        patch("codex.config.get_settings") as mock_cfg,
-    ):
-        mock_cfg.return_value.wiki_dir = tmpdir
-        result = wiki_read("nonexistent_concept")
-
-    assert result == {"error": "feature not available"}
-
-
-def test_wiki_read_existing_page_returns_markdown() -> None:
-    """wiki_read() with an existing page must return {markdown, sources}."""
-    from codex.mcp_server import wiki_read
-
-    with (
-        tempfile.TemporaryDirectory() as tmpdir,
-        patch("codex.config.get_settings") as mock_cfg,
-    ):
-        mock_cfg.return_value.wiki_dir = tmpdir
-        # Write a fake wiki page
-        page = Path(tmpdir) / "conformal_maps.md"
-        page.write_text("# Conformal Maps\n\nSome content.", encoding="utf-8")
-        result = wiki_read("conformal_maps")
-
-    assert "markdown" in result
-    assert "sources" in result
-    assert "# Conformal Maps" in str(result["markdown"])
-
-
-def test_wiki_read_exception_returns_not_available() -> None:
-    """wiki_read() must degrade gracefully when get_settings raises."""
-    from codex.mcp_server import wiki_read
-
-    with patch("codex.config.get_settings", side_effect=RuntimeError("config error")):
-        result = wiki_read("any_slug")
-
-    assert result == {"error": "feature not available"}
-
-
-# ---------------------------------------------------------------------------
-# wiki_list — missing wiki dir → empty list
-# ---------------------------------------------------------------------------
-
-
-def test_wiki_list_missing_dir_returns_empty() -> None:
-    """wiki_list() must return [] when the wiki directory does not exist."""
-    from codex.mcp_server import wiki_list
-
-    with (
-        tempfile.TemporaryDirectory() as tmpdir,
-        patch("codex.config.get_settings") as mock_cfg,
-    ):
-        mock_cfg.return_value.wiki_dir = str(Path(tmpdir) / "no_wiki_here")
-        result = wiki_list()
-
-    assert result == []
-
-
-def test_wiki_list_existing_dir_returns_pages() -> None:
-    """wiki_list() must return page entries for .md files in the wiki dir."""
-    from codex.mcp_server import wiki_list
-
-    with (
-        tempfile.TemporaryDirectory() as tmpdir,
-        patch("codex.config.get_settings") as mock_cfg,
-    ):
-        mock_cfg.return_value.wiki_dir = tmpdir
-        (Path(tmpdir) / "concept_a.md").write_text("# A", encoding="utf-8")
-        (Path(tmpdir) / "concept_b.md").write_text("# B", encoding="utf-8")
-        (Path(tmpdir) / "index.md").write_text("# Index", encoding="utf-8")  # must be skipped
-        result = wiki_list()
-
-    slugs = {r["slug"] for r in result}
-    assert "concept_a" in slugs
-    assert "concept_b" in slugs
-    assert "index" not in slugs  # index.md is excluded
-
-
-# ---------------------------------------------------------------------------
-# discover_leads — domain function raises → error dict, no crash
-# ---------------------------------------------------------------------------
-
-
-def test_discover_leads_db_error_returns_error() -> None:
-    """discover_leads() must return [{"error": ...}] when the discovery call fails."""
-    from codex.mcp_server import discover_leads
-
-    # Patch the domain function directly so the exception always surfaces
-    with patch("codex.discover.discovery_leads", side_effect=RuntimeError("no DB")):
-        result = discover_leads(limit=5)
-
-    assert len(result) == 1
-    assert "error" in result[0]
-
-
-# ---------------------------------------------------------------------------
-# synthesis_browse — missing leads dir → "feature not available"
-# ---------------------------------------------------------------------------
-
-
-def test_synthesis_browse_missing_dir_returns_not_available() -> None:
-    """synthesis_browse() must return [{'error': 'feature not available'}] when leads/ absent."""
-    from codex.mcp_server import synthesis_browse
-
-    with (
-        tempfile.TemporaryDirectory() as tmpdir,
-        patch("codex.config.get_settings") as mock_cfg,
-    ):
-        mock_cfg.return_value.leads_dir = str(Path(tmpdir) / "no_leads_here")
-        result = synthesis_browse()
-
-    assert result == [{"error": "feature not available"}]
-
-
-def test_synthesis_browse_existing_dir_returns_items() -> None:
-    """synthesis_browse() must parse and return JSON files from the leads dir."""
-    from codex.mcp_server import synthesis_browse
-
-    with (
-        tempfile.TemporaryDirectory() as tmpdir,
-        patch("codex.config.get_settings") as mock_cfg,
-    ):
-        leads_dir = Path(tmpdir) / "leads"
-        leads_dir.mkdir()
-        (leads_dir / "lead1.json").write_text(
-            json.dumps({"kind": "connection", "title": "C1"}), encoding="utf-8"
-        )
-        (leads_dir / "lead2.json").write_text(
-            json.dumps({"kind": "conjecture", "title": "C2"}), encoding="utf-8"
-        )
-        mock_cfg.return_value.leads_dir = str(leads_dir)
-        result = synthesis_browse(kind="all")
-
-    assert len(result) == 2
-    kinds = {str(r.get("kind")) for r in result}
-    assert "connection" in kinds
-    assert "conjecture" in kinds
-
-
-def test_synthesis_browse_kind_filter() -> None:
-    """synthesis_browse(kind='conjecture') must return only conjectures."""
-    from codex.mcp_server import synthesis_browse
-
-    with (
-        tempfile.TemporaryDirectory() as tmpdir,
-        patch("codex.config.get_settings") as mock_cfg,
-    ):
-        leads_dir = Path(tmpdir) / "leads"
-        leads_dir.mkdir()
-        (leads_dir / "lead1.json").write_text(
-            json.dumps({"kind": "connection", "title": "C1"}), encoding="utf-8"
-        )
-        (leads_dir / "lead2.json").write_text(
-            json.dumps({"kind": "conjecture", "title": "C2"}), encoding="utf-8"
-        )
-        mock_cfg.return_value.leads_dir = str(leads_dir)
-        result = synthesis_browse(kind="conjecture")
-
-    assert all(str(r.get("kind")) == "conjecture" for r in result)
-    assert len(result) == 1
-
-
-# ---------------------------------------------------------------------------
-# provenance_verify — exception → "feature not available"
-# ---------------------------------------------------------------------------
-
-
-def test_provenance_verify_exception_returns_not_available() -> None:
-    """provenance_verify() must return [{"error": ...}] on scan failure."""
-    from codex.mcp_server import provenance_verify
-
-    with patch("codex.provenance.scan_cite_tags", side_effect=RuntimeError("scan failed")):
-        result = provenance_verify("/some/nonexistent/path")
-
-    assert len(result) == 1
-    assert "error" in result[0]

tests/mcp/test_http_auth.py

@@ -1,122 +0,0 @@
-"""test_http_auth.py — HTTP transport token-auth enforcement.
-
-Tests:
-  1. _require_http_token() raises RuntimeError when MCP_AUTH_TOKEN is absent.
-  2. _require_http_token() returns the token when it is set.
-  3. main() with transport='http' raises without token (refuses to start).
-  4. main() with transport='stdio' runs mcp.run (happy path, no token needed).
-"""
-
-from __future__ import annotations
-
-from unittest.mock import MagicMock, patch
-
-import pytest
-from pydantic import SecretStr
-
-# ---------------------------------------------------------------------------
-# _require_http_token
-# ---------------------------------------------------------------------------
-
-
-def test_require_http_token_raises_without_token() -> None:
-    """_require_http_token() must raise RuntimeError when mcp_auth_token is None."""
-    from codex.mcp_server import _require_http_token
-
-    mock_settings = MagicMock()
-    mock_settings.mcp_auth_token = None
-
-    with (
-        patch("codex.config.get_settings", return_value=mock_settings),
-        pytest.raises(RuntimeError, match="MCP_AUTH_TOKEN"),
-    ):
-        _require_http_token()
-
-
-def test_require_http_token_returns_token_when_set() -> None:
-    """_require_http_token() must return the token string when it is configured."""
-    from codex.mcp_server import _require_http_token
-
-    mock_settings = MagicMock()
-    mock_settings.mcp_auth_token = SecretStr("super-secret-token")
-
-    with patch("codex.config.get_settings", return_value=mock_settings):
-        result = _require_http_token()
-
-    assert result == "super-secret-token"
-
-
-# ---------------------------------------------------------------------------
-# main() — HTTP transport without token → RuntimeError
-# ---------------------------------------------------------------------------
-
-
-def test_main_http_without_token_raises() -> None:
-    """main() with MCP_TRANSPORT=http and no token must raise RuntimeError."""
-    from codex.mcp_server import main
-
-    mock_settings = MagicMock()
-    mock_settings.mcp_transport = "http"
-    mock_settings.mcp_auth_token = None
-
-    with (
-        patch("codex.config.get_settings", return_value=mock_settings),
-        pytest.raises(RuntimeError, match="MCP_AUTH_TOKEN"),
-    ):
-        main()
-
-
-# ---------------------------------------------------------------------------
-# main() — stdio transport runs mcp.run
-# ---------------------------------------------------------------------------
-
-
-def test_main_stdio_calls_mcp_run() -> None:
-    """main() with stdio transport must call mcp.run(transport='stdio')."""
-    from codex import mcp_server
-
-    mock_settings = MagicMock()
-    mock_settings.mcp_transport = "stdio"
-
-    with (
-        patch("codex.config.get_settings", return_value=mock_settings),
-        patch.object(mcp_server.mcp, "run") as mock_run,
-    ):
-        mcp_server.main()
-
-    mock_run.assert_called_once_with(transport="stdio")
-
-
-# ---------------------------------------------------------------------------
-# main() — HTTP transport with valid token attempts uvicorn.run
-# ---------------------------------------------------------------------------
-
-
-def test_main_http_with_token_calls_uvicorn() -> None:
-    """main() with http transport + valid token must invoke uvicorn.run."""
-    from codex import mcp_server
-
-    mock_settings = MagicMock()
-    mock_settings.mcp_transport = "http"
-    mock_settings.mcp_auth_token = SecretStr("test-token")
-    mock_settings.mcp_host = "127.0.0.1"
-    mock_settings.mcp_port = 8765
-
-    # Stub the ASGI app returned by streamable_http_app
-    mock_app = MagicMock()
-    mock_app.add_middleware = MagicMock()
-
-    with (
-        patch("codex.config.get_settings", return_value=mock_settings),
-        patch.object(mcp_server.mcp, "streamable_http_app", return_value=mock_app),
-        patch("uvicorn.run") as mock_uvicorn,
-    ):
-        mcp_server.main()
-
-    mock_uvicorn.assert_called_once()
-    call_kwargs = mock_uvicorn.call_args
-    assert call_kwargs is not None
-    # host and port must match settings
-    _, kwargs = call_kwargs
-    assert kwargs.get("host") == "127.0.0.1"
-    assert kwargs.get("port") == 8765

tests/mcp/test_readonly.py

@@ -1,83 +0,0 @@
-"""test_readonly.py — CRITICAL: ensure no write/ingest tools are exposed via MCP.
-
-This test MUST fail if any write or ingest tool is accidentally registered.
-It is the primary enforcement point for the read-only invariant of F-14.
-
-Read-only tools (allowed):
-  search, ask, wiki_read, wiki_list, discover_leads,
-  provenance_verify, synthesis_browse
-
-Write/ingest tools that must NEVER appear:
-  Any tool whose name contains an ingest/write/delete/add/update/remove
-  verb, or any of the known write-mode CLI operations.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import re
-
-# ---------------------------------------------------------------------------
-# Write-verb pattern — any tool matching this is a forbidden write tool.
-# ---------------------------------------------------------------------------
-
-_WRITE_VERB_RE = re.compile(
-    r"(?:^|_)(ingest|write|delete|remove|add|update|upsert|insert|create|sync|compile|export)",
-    re.IGNORECASE,
-)
-
-# Explicit allowlist — only these names may appear.
-ALLOWED_TOOLS = frozenset(
-    {
-        "search",
-        "ask",
-        "wiki_read",
-        "wiki_list",
-        "discover_leads",
-        "provenance_verify",
-        "synthesis_browse",
-    }
-)
-
-
-def test_no_write_tools_registered() -> None:
-    """CRITICAL: fail if any write-verb tool is registered on the MCP server."""
-    from codex.mcp_server import mcp
-
-    tools = asyncio.run(mcp.list_tools())
-    registered = {t.name for t in tools}
-
-    violations: list[str] = []
-    for name in registered:
-        if _WRITE_VERB_RE.search(name):
-            violations.append(name)
-
-    assert not violations, (
-        f"Write/ingest tools found in MCP server (FORBIDDEN): {violations}. "
-        "MCP server must expose read-only tools only."
-    )
-
-
-def test_only_allowed_tools_registered() -> None:
-    """CRITICAL: registered tools must be a subset of the explicit allowlist."""
-    from codex.mcp_server import mcp
-
-    tools = asyncio.run(mcp.list_tools())
-    registered = {t.name for t in tools}
-
-    disallowed = registered - ALLOWED_TOOLS
-    assert not disallowed, (
-        f"Tools not on the read-only allowlist: {disallowed}. "
-        "Add them to ALLOWED_TOOLS only after verifying they are read-only."
-    )
-
-
-def test_all_allowed_tools_present() -> None:
-    """Every tool in the allowlist must be registered (completeness check)."""
-    from codex.mcp_server import mcp
-
-    tools = asyncio.run(mcp.list_tools())
-    registered = {t.name for t in tools}
-
-    missing = ALLOWED_TOOLS - registered
-    assert not missing, f"Expected read-only tools missing from server: {missing}"

tests/mcp/test_search.py

@@ -1,176 +0,0 @@
-"""test_search.py — verify the search tool: retrieval, return format, error handling.
-
-Since search() imports get_embedder and get_conn lazily (inside the function body),
-we patch them at their source modules: codex.embed and codex.db.
-"""
-
-from __future__ import annotations
-
-from contextlib import contextmanager
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-import numpy as np
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _make_conn_cm(mock_conn: MagicMock) -> Any:
-    """Return a context-manager factory that yields *mock_conn*."""
-
-    @contextmanager  # type: ignore[arg-type]
-    def _cm() -> Any:
-        yield mock_conn
-
-    return _cm
-
-
-def _fake_row(
-    bibkey: str,
-    paper_id: str,
-    ord_val: int,
-    content: str,
-    dist: float,
-) -> dict[str, Any]:
-    return {
-        "bibkey": bibkey,
-        "paper_id": paper_id,
-        "ord": ord_val,
-        "content": content,
-        "dist": dist,
-    }
-
-
-# ---------------------------------------------------------------------------
-# Tests
-# ---------------------------------------------------------------------------
-
-
-def test_search_returns_correct_keys() -> None:
-    """search() must return dicts with the required five keys."""
-    from codex.mcp_server import search
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _fake_row("Springborn2008", "springborn-2008", 16, "The volume formula is V = ...", 0.2),
-    ]
-
-    fake_dense = np.zeros((1, 1024), dtype=np.float32)
-    mock_embedder = MagicMock()
-    mock_embedder.encode_dense.return_value = fake_dense
-
-    with (
-        patch("codex.embed.get_embedder", return_value=mock_embedder),
-        patch("codex.db.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        results = search("volume formula", limit=5)
-
-    assert len(results) == 1
-    hit = results[0]
-    assert set(hit.keys()) == {"bibkey", "paper_id", "locator", "score", "snippet"}
-
-
-def test_search_score_is_similarity_not_distance() -> None:
-    """score = 1 - distance, so a dist of 0.0 should yield score 1.0."""
-    from codex.mcp_server import search
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _fake_row("Author2020", "paper-x", 3, "Some chunk content here.", 0.0),
-    ]
-
-    fake_dense = np.zeros((1, 1024), dtype=np.float32)
-    mock_embedder = MagicMock()
-    mock_embedder.encode_dense.return_value = fake_dense
-
-    with (
-        patch("codex.embed.get_embedder", return_value=mock_embedder),
-        patch("codex.db.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        results = search("some query")
-
-    assert results[0]["score"] == 1.0
-
-
-def test_search_locator_format() -> None:
-    """locator must be 'chunk <ord>'."""
-    from codex.mcp_server import search
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _fake_row("Auth2021", "paper-y", 42, "Content.", 0.3),
-    ]
-
-    fake_dense = np.zeros((1, 1024), dtype=np.float32)
-    mock_embedder = MagicMock()
-    mock_embedder.encode_dense.return_value = fake_dense
-
-    with (
-        patch("codex.embed.get_embedder", return_value=mock_embedder),
-        patch("codex.db.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        results = search("query")
-
-    assert results[0]["locator"] == "chunk 42"
-
-
-def test_search_snippet_truncated_to_300() -> None:
-    """snippet must be truncated to 300 characters."""
-    from codex.mcp_server import search
-
-    long_content = "x" * 500
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _fake_row("Auth2022", "paper-z", 1, long_content, 0.1),
-    ]
-
-    fake_dense = np.zeros((1, 1024), dtype=np.float32)
-    mock_embedder = MagicMock()
-    mock_embedder.encode_dense.return_value = fake_dense
-
-    with (
-        patch("codex.embed.get_embedder", return_value=mock_embedder),
-        patch("codex.db.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        results = search("query")
-
-    assert len(results[0]["snippet"]) == 300  # type: ignore[arg-type]
-
-
-def test_search_returns_error_dict_on_exception() -> None:
-    """When the embedder raises, search must return [{"error": ...}], not crash."""
-    from codex.mcp_server import search
-
-    mock_embedder = MagicMock()
-    mock_embedder.encode_dense.side_effect = RuntimeError("DB down")
-
-    with patch("codex.embed.get_embedder", return_value=mock_embedder):
-        results = search("query")
-
-    assert len(results) == 1
-    assert "error" in results[0]
-
-
-def test_search_provenance_keys() -> None:
-    """Each hit must carry bibkey and paper_id for provenance traceability."""
-    from codex.mcp_server import search
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _fake_row("Crane1999", "crane-1999", 7, "Discrete exterior calculus ...", 0.15),
-    ]
-
-    fake_dense = np.zeros((1, 1024), dtype=np.float32)
-    mock_embedder = MagicMock()
-    mock_embedder.encode_dense.return_value = fake_dense
-
-    with (
-        patch("codex.embed.get_embedder", return_value=mock_embedder),
-        patch("codex.db.get_conn", side_effect=_make_conn_cm(mock_conn)),
-    ):
-        results = search("exterior calculus")
-
-    assert results[0]["bibkey"] == "Crane1999"
-    assert results[0]["paper_id"] == "crane-1999"

tests/mcp/test_server.py

@@ -1,49 +0,0 @@
-"""test_server.py — verify the MCP server instantiates and all tools are registered."""
-
-from __future__ import annotations
-
-import asyncio
-
-# ---------------------------------------------------------------------------
-# Expected tools (must match EXACTLY what mcp_server.py registers)
-# ---------------------------------------------------------------------------
-
-EXPECTED_TOOLS = {
-    "search",
-    "ask",
-    "wiki_read",
-    "wiki_list",
-    "discover_leads",
-    "provenance_verify",
-    "synthesis_browse",
-}
-
-
-def test_server_instance_exists() -> None:
-    """The module-level ``mcp`` FastMCP instance must exist."""
-    from codex.mcp_server import mcp
-
-    assert mcp is not None
-    assert mcp.name == "codex"
-
-
-def test_all_tools_registered() -> None:
-    """Every expected tool name must be registered on the server."""
-    from codex.mcp_server import mcp
-
-    tools = asyncio.run(mcp.list_tools())
-    registered = {t.name for t in tools}
-
-    missing = EXPECTED_TOOLS - registered
-    assert not missing, f"Tools not registered: {missing}"
-
-
-def test_no_extra_unexpected_write_tools() -> None:
-    """No tool outside the expected set should be registered (belt-and-suspenders)."""
-    from codex.mcp_server import mcp
-
-    tools = asyncio.run(mcp.list_tools())
-    registered = {t.name for t in tools}
-
-    unexpected = registered - EXPECTED_TOOLS
-    assert not unexpected, f"Unexpected tools registered: {unexpected}"

tests/parsing/test_figures.py

@@ -1,245 +0,0 @@
-"""Tests for codex.parsing.figures — figure extraction module.
-
-All external dependencies (fitz, PIL) are mocked so the suite runs offline.
-"""
-
-from __future__ import annotations
-
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-
-def _make_mock_rect(
-    x0: float = 10.0, y0: float = 10.0, x1: float = 200.0, y1: float = 150.0
-) -> MagicMock:
-    """Build a mock fitz.Rect-like object."""
-    rect = MagicMock()
-    rect.x0 = x0
-    rect.y0 = y0
-    rect.x1 = x1
-    rect.y1 = y1
-    return rect
-
-
-def _make_mock_doc(
-    images: list[tuple[int, ...]],  # list of (xref, ...)
-    text_blocks: list[Any],
-    img_rects: list[Any],
-    page_count: int = 1,
-) -> tuple[MagicMock, MagicMock]:
-    """Build a mock fitz document with one page."""
-    mock_pix = MagicMock()
-    mock_pix.n = 3  # RGB — no conversion needed
-    mock_pix.save = MagicMock()
-
-    mock_page = MagicMock()
-    mock_page.get_images.return_value = images
-    mock_page.get_text.return_value = text_blocks
-    mock_page.get_image_rects.return_value = img_rects
-
-    mock_doc = MagicMock()
-    mock_doc.__len__.return_value = page_count
-    mock_doc.__getitem__ = MagicMock(return_value=mock_page)
-
-    return mock_doc, mock_pix
-
-
-class TestExtractFigures:
-    """Tests for extract_figures()."""
-
-    def test_no_caption_returns_empty_string(self, tmp_path: Any) -> None:
-        """Figure with no nearby caption gets empty caption string."""
-        img_rect = _make_mock_rect(10, 10, 200, 150)
-        text_blocks = [
-            (300.0, 300.0, 400.0, 320.0, "Unrelated text here.", 0, 0),
-        ]
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=text_blocks,
-            img_rects=[img_rect],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert len(results) == 1
-        assert results[0].caption == ""
-
-    def test_figure_png_is_saved(self, tmp_path: Any) -> None:
-        """PNG save is called with the correct output path."""
-        img_rect = _make_mock_rect()
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=[],
-            img_rects=[img_rect],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert len(results) == 1
-        mock_pix.save.assert_called_once()
-        saved_path: str = mock_pix.save.call_args[0][0]
-        assert saved_path.endswith(".png")
-
-    def test_caption_detected_below_image(self, tmp_path: Any) -> None:
-        """Caption block immediately below the image is detected."""
-        img_rect = _make_mock_rect(10, 10, 200, 150)
-        caption_block = (10.0, 155.0, 200.0, 170.0, "Figure 1: Architecture overview.", 0, 0)
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=[caption_block],
-            img_rects=[img_rect],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert results[0].caption == "Figure 1: Architecture overview."
-
-    def test_caption_detected_above_image(self, tmp_path: Any) -> None:
-        """Caption block immediately above the image is also detected."""
-        img_rect = _make_mock_rect(10, 80, 200, 200)
-        caption_block = (10.0, 50.0, 200.0, 75.0, "Fig. 2: Loss curve.", 0, 0)
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=[caption_block],
-            img_rects=[img_rect],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert results[0].caption == "Fig. 2: Loss curve."
-
-    def test_german_caption_prefix(self, tmp_path: Any) -> None:
-        """'Abbildung' prefix is recognized as a valid German caption."""
-        img_rect = _make_mock_rect(10, 10, 200, 150)
-        caption_block = (10.0, 155.0, 200.0, 170.0, "Abbildung 3: Ergebnisse.", 0, 0)
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=[caption_block],
-            img_rects=[img_rect],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert "Abbildung" in results[0].caption
-
-    def test_unrelated_text_not_used_as_caption(self, tmp_path: Any) -> None:
-        """Text blocks not starting with a caption prefix are ignored."""
-        img_rect = _make_mock_rect(10, 10, 200, 150)
-        non_caption = (10.0, 155.0, 200.0, 170.0, "This is a random sentence.", 0, 0)
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=[non_caption],
-            img_rects=[img_rect],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert results[0].caption == ""
-
-    def test_no_images_returns_empty_list(self, tmp_path: Any) -> None:
-        """PDF with no embedded images returns empty list."""
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[],
-            text_blocks=[],
-            img_rects=[],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert results == []
-
-    def test_image_without_rect_is_skipped(self, tmp_path: Any) -> None:
-        """Images for which get_image_rects returns empty list are skipped."""
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=[],
-            img_rects=[],  # no rect
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("paper.pdf", output_dir=str(tmp_path))
-
-        assert results == []
-
-    def test_output_dir_created_if_absent(self, tmp_path: Any) -> None:
-        """output_dir is created if it does not exist."""
-        new_dir = tmp_path / "new_subdir" / "figures"
-        assert not new_dir.exists()
-
-        mock_doc, mock_pix = _make_mock_doc(images=[], text_blocks=[], img_rects=[])
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            extract_figures("paper.pdf", output_dir=str(new_dir))
-
-        assert new_dir.exists()
-
-    def test_paper_id_from_pdf_stem(self, tmp_path: Any) -> None:
-        """paper_id in FigureChunk is the stem of the PDF filename."""
-        img_rect = _make_mock_rect()
-        mock_doc, mock_pix = _make_mock_doc(
-            images=[(5, 0, 0, 0, 0, 0, 0, 0, 8, "png")],
-            text_blocks=[],
-            img_rects=[img_rect],
-        )
-
-        with (
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Pixmap", return_value=mock_pix),
-        ):
-            from codex.parsing.figures import extract_figures
-
-            results = extract_figures("/tmp/2301.07041.pdf", output_dir=str(tmp_path))
-
-        assert results[0].paper_id == "2301.07041"

tests/parsing/test_grobid.py

@@ -1,161 +0,0 @@
-"""Tests for codex.parsing.grobid."""
-
-from __future__ import annotations
-
-from pathlib import Path
-from unittest.mock import MagicMock, patch
-
-from codex.parsing.grobid import extract_references, extract_structure
-
-# Real GROBID TEI output uses type="arXiv" (camel-case), not "arxiv".
-_TEI_XML = """\
-<?xml version="1.0" encoding="UTF-8"?>
-<TEI xmlns="http://www.tei-c.org/ns/1.0">
-  <text>
-    <back>
-      <div type="references">
-        <listBibl>
-          <biblStruct>
-            <analytic>
-              <title level="a">Deep Learning for NLP</title>
-              <author>
-                <persName>
-                  <forename>Jane</forename>
-                  <surname>Doe</surname>
-                </persName>
-              </author>
-            </analytic>
-            <monogr>
-              <imprint>
-                <date type="published" when="2021-06"/>
-              </imprint>
-            </monogr>
-            <idno type="DOI">10.1234/nlp.2021</idno>
-            <idno type="arXiv">2101.00001</idno>
-          </biblStruct>
-          <biblStruct>
-            <analytic>
-              <title level="a">Attention Is All You Need</title>
-              <author>
-                <persName>
-                  <forename>Ashish</forename>
-                  <surname>Vaswani</surname>
-                </persName>
-              </author>
-              <author>
-                <persName>
-                  <forename>Noam</forename>
-                  <surname>Shazeer</surname>
-                </persName>
-              </author>
-            </analytic>
-            <monogr>
-              <imprint>
-                <date type="published" when="2017"/>
-              </imprint>
-            </monogr>
-            <idno type="DOI">10.5678/attention</idno>
-          </biblStruct>
-        </listBibl>
-      </div>
-    </back>
-  </text>
-</TEI>
-"""
-
-
-class _FakeResponse:
-    text = _TEI_XML
-    status_code = 200
-
-    def raise_for_status(self) -> None:
-        pass
-
-
-def _mock_post(xml: str = _TEI_XML) -> MagicMock:
-    fake = MagicMock()
-    fake.text = xml
-    fake.status_code = 200
-    fake.raise_for_status = MagicMock()
-    return fake
-
-
-class TestExtractReferences:
-    def test_returns_two_dicts(self, tmp_path: Path) -> None:
-        pdf = tmp_path / "paper.pdf"
-        pdf.write_bytes(b"%PDF fake")
-
-        with patch("httpx.Client") as mock_client_cls:
-            mock_client = MagicMock()
-            mock_client_cls.return_value.__enter__.return_value = mock_client
-            mock_client.post.return_value = _mock_post()
-
-            refs = extract_references(str(pdf), grobid_url="http://localhost:8070")
-
-        assert len(refs) == 2
-
-    def test_all_keys_present(self, tmp_path: Path) -> None:
-        pdf = tmp_path / "paper.pdf"
-        pdf.write_bytes(b"%PDF fake")
-
-        with patch("httpx.Client") as mock_client_cls:
-            mock_client = MagicMock()
-            mock_client_cls.return_value.__enter__.return_value = mock_client
-            mock_client.post.return_value = _mock_post()
-
-            refs = extract_references(str(pdf), grobid_url="http://localhost:8070")
-
-        required_keys = {"title", "authors", "year", "doi", "arxiv_id"}
-        for ref in refs:
-            assert required_keys == set(ref.keys()), f"Missing keys in {ref}"
-
-    def test_first_ref_values(self, tmp_path: Path) -> None:
-        pdf = tmp_path / "paper.pdf"
-        pdf.write_bytes(b"%PDF fake")
-
-        with patch("httpx.Client") as mock_client_cls:
-            mock_client = MagicMock()
-            mock_client_cls.return_value.__enter__.return_value = mock_client
-            mock_client.post.return_value = _mock_post()
-
-            refs = extract_references(str(pdf), grobid_url="http://localhost:8070")
-
-        first = refs[0]
-        assert first["title"] == "Deep Learning for NLP"
-        assert "Jane Doe" in first["authors"]
-        assert first["year"] == "2021"
-        assert first["doi"] == "10.1234/nlp.2021"
-        assert first["arxiv_id"] == "2101.00001"
-
-    def test_second_ref_missing_arxiv_is_empty_string(self, tmp_path: Path) -> None:
-        pdf = tmp_path / "paper.pdf"
-        pdf.write_bytes(b"%PDF fake")
-
-        with patch("httpx.Client") as mock_client_cls:
-            mock_client = MagicMock()
-            mock_client_cls.return_value.__enter__.return_value = mock_client
-            mock_client.post.return_value = _mock_post()
-
-            refs = extract_references(str(pdf), grobid_url="http://localhost:8070")
-
-        second = refs[1]
-        assert second["arxiv_id"] == ""
-        assert second["title"] == "Attention Is All You Need"
-        assert second["year"] == "2017"
-
-
-class TestExtractStructure:
-    def test_returns_xml_string(self, tmp_path: Path) -> None:
-        pdf = tmp_path / "paper.pdf"
-        pdf.write_bytes(b"%PDF fake")
-
-        with patch("httpx.Client") as mock_client_cls:
-            mock_client = MagicMock()
-            mock_client_cls.return_value.__enter__.return_value = mock_client
-            mock_client.post.return_value = _mock_post("<TEI>raw xml</TEI>")
-
-            result = extract_structure(str(pdf), grobid_url="http://localhost:8070")
-
-        assert result == "<TEI>raw xml</TEI>"
-        mock_client.post.assert_called_once()
-        assert "processFulltextDocument" in mock_client.post.call_args[0][0]

tests/parsing/test_mathpix.py

@@ -1,387 +0,0 @@
-"""Tests for codex.parsing.mathpix — formula extraction module.
-
-All external dependencies (fitz, pix2tex, httpx, get_settings) are mocked
-so the suite runs offline without real API calls or model downloads.
-"""
-
-from __future__ import annotations
-
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-import pytest
-from pydantic import SecretStr
-
-from codex.models import FormulaChunk
-
-
-def _mock_open_pdf() -> MagicMock:
-    """Return a mock that satisfies ``open(path, 'rb')`` as a context manager."""
-    mock_file = MagicMock()
-    mock_file.__enter__ = MagicMock(return_value=mock_file)
-    mock_file.__exit__ = MagicMock(return_value=False)
-    mock_open = MagicMock(return_value=mock_file)
-    return mock_open
-
-
-class TestMathPixAPIPath:
-    """Tests for the MathPix cloud API backend."""
-
-    def test_mathpix_200_returns_formulas(self) -> None:
-        """extract_formulas routes to MathPix when creds are provided and parses result."""
-        mmd_content = "\\[\n\\alpha + \\beta\n\\]\n\\[\nE = mc^2\n\\]"
-
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = None
-        mock_settings.mathpix_app_key = None
-        mock_settings.pix2tex_fallback = True
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch("builtins.open", _mock_open_pdf()),
-            patch("httpx.post") as mock_post,
-            patch("httpx.get") as mock_get,
-            patch("time.sleep"),
-        ):
-            mock_post.return_value = MagicMock(
-                status_code=200,
-                json=lambda: {"pdf_id": "abc123"},
-            )
-            mock_post.return_value.raise_for_status = MagicMock()
-
-            mock_get.return_value = MagicMock(
-                status_code=200,
-                text=mmd_content,
-            )
-
-            from codex.parsing.mathpix import _extract_formulas_mathpix
-
-            results = _extract_formulas_mathpix("paper.pdf", "app_id", "app_key")
-
-        assert len(results) == 2
-        assert all(isinstance(r, FormulaChunk) for r in results)
-        assert results[0].raw_latex == "\\[\n\\alpha + \\beta\n\\]"
-
-    def test_mathpix_empty_pdf_id_returns_empty(self) -> None:
-        """When MathPix returns no pdf_id, result is empty list."""
-        with (
-            patch("builtins.open", _mock_open_pdf()),
-            patch("httpx.post") as mock_post,
-            patch("time.sleep"),
-        ):
-            mock_post.return_value = MagicMock(
-                status_code=200,
-                json=lambda: {},  # No pdf_id
-            )
-            mock_post.return_value.raise_for_status = MagicMock()
-
-            from codex.parsing.mathpix import _extract_formulas_mathpix
-
-            results = _extract_formulas_mathpix("paper.pdf", "app_id", "app_key")
-
-        assert results == []
-
-    def test_mathpix_paper_id_from_stem(self) -> None:
-        """paper_id in FormulaChunk is derived from the PDF filename stem."""
-        mmd_content = "\\[\nx^2\n\\]"
-
-        with (
-            patch("builtins.open", _mock_open_pdf()),
-            patch("httpx.post") as mock_post,
-            patch("httpx.get") as mock_get,
-            patch("time.sleep"),
-        ):
-            mock_post.return_value = MagicMock(
-                status_code=200,
-                json=lambda: {"pdf_id": "xyz"},
-            )
-            mock_post.return_value.raise_for_status = MagicMock()
-            mock_get.return_value = MagicMock(status_code=200, text=mmd_content)
-
-            from codex.parsing.mathpix import _extract_formulas_mathpix
-
-            results = _extract_formulas_mathpix("/tmp/2301.07041.pdf", "id", "key")
-
-        assert results[0].paper_id == "2301.07041"
-
-    def test_mathpix_page_counter_increments(self) -> None:
-        """Page counter increments on MathPix page markers in MMD."""
-        mmd_content = "\\[\nA\n\\]\n<!-- Page 2 -->\n\\[\nB\n\\]"
-
-        with (
-            patch("builtins.open", _mock_open_pdf()),
-            patch("httpx.post") as mock_post,
-            patch("httpx.get") as mock_get,
-            patch("time.sleep"),
-        ):
-            mock_post.return_value = MagicMock(
-                status_code=200,
-                json=lambda: {"pdf_id": "p1"},
-            )
-            mock_post.return_value.raise_for_status = MagicMock()
-            mock_get.return_value = MagicMock(status_code=200, text=mmd_content)
-
-            from codex.parsing.mathpix import _extract_formulas_mathpix
-
-            results = _extract_formulas_mathpix("p.pdf", "id", "key")
-
-        assert results[0].page == 1
-        assert results[1].page == 2
-
-    def test_mathpix_http_error_propagates(self) -> None:
-        """HTTP errors from MathPix are raised (tenacity reraises after retries)."""
-        import httpx
-
-        with (
-            patch("builtins.open", _mock_open_pdf()),
-            patch("httpx.post") as mock_post,
-        ):
-            mock_post.return_value = MagicMock(status_code=401)
-            mock_post.return_value.raise_for_status.side_effect = httpx.HTTPStatusError(
-                "401", request=MagicMock(), response=MagicMock()
-            )
-
-            from codex.parsing.mathpix import _extract_formulas_mathpix
-
-            with pytest.raises(httpx.HTTPStatusError):
-                _extract_formulas_mathpix("p.pdf", "id", "key")
-
-
-class TestPix2TexFallbackPath:
-    """Tests for the local pix2tex backend."""
-
-    def _make_fitz_page(
-        self,
-        blocks: list[Any],
-        pixmap_samples: bytes = b"\xff" * (30 * 100 * 3),
-        pixmap_w: int = 100,
-        pixmap_h: int = 30,
-    ) -> MagicMock:
-        """Build a mock fitz page with given text blocks and a fake pixmap."""
-        mock_pix = MagicMock()
-        mock_pix.width = pixmap_w
-        mock_pix.height = pixmap_h
-        mock_pix.samples = pixmap_samples
-
-        mock_page = MagicMock()
-        mock_page.get_text.return_value = blocks
-        mock_page.get_pixmap.return_value = mock_pix
-        return mock_page
-
-    def test_pix2tex_fallback_invoked_without_creds(self) -> None:
-        """When no MathPix creds, pix2tex fallback is called."""
-        math_block = (0.0, 0.0, 200.0, 50.0, "∑ α∫β∇γ∂δ∞ε", 0, 0)
-
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = None
-        mock_settings.mathpix_app_key = None
-        mock_settings.pix2tex_fallback = True
-        mock_settings.figures_dir = "/tmp/figs"
-
-        mock_model = MagicMock(return_value=r"\sum \alpha")
-
-        mock_page = self._make_fitz_page([math_block])
-        mock_doc = MagicMock()
-        mock_doc.__len__.return_value = 1
-        mock_doc.__getitem__ = MagicMock(return_value=mock_page)
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch("codex.parsing.mathpix._get_pix2tex_model", return_value=mock_model),
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Matrix"),
-            patch("fitz.Rect"),
-        ):
-            from PIL import Image
-
-            with patch.object(Image, "frombytes", return_value=MagicMock()):
-                from codex.parsing.mathpix import extract_formulas
-
-                results = extract_formulas("paper.pdf")
-
-        assert len(results) == 1
-        assert results[0].raw_latex == r"\sum \alpha"
-
-    def test_pix2tex_bbox_size_filter(self) -> None:
-        """Blocks smaller than MIN_HEIGHT_PT or MIN_WIDTH_PT are skipped."""
-        tiny_block = (0.0, 0.0, 10.0, 5.0, "∑∫∂∞∇ε", 0, 0)  # 10pt wide, 5pt tall
-
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = None
-        mock_settings.mathpix_app_key = None
-        mock_settings.pix2tex_fallback = True
-
-        mock_model = MagicMock(return_value=r"\sum")
-        mock_page = self._make_fitz_page([tiny_block])
-        mock_doc = MagicMock()
-        mock_doc.__len__.return_value = 1
-        mock_doc.__getitem__ = MagicMock(return_value=mock_page)
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch("codex.parsing.mathpix._get_pix2tex_model", return_value=mock_model),
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Matrix"),
-            patch("fitz.Rect"),
-        ):
-            from codex.parsing.mathpix import extract_formulas
-
-            results = extract_formulas("paper.pdf")
-
-        assert results == []
-
-    def test_pix2tex_math_ratio_filter(self) -> None:
-        """Blocks with low math-character ratio are skipped."""
-        plain_text_block = (0.0, 0.0, 200.0, 50.0, "This is just regular text with no math.", 0, 0)
-
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = None
-        mock_settings.mathpix_app_key = None
-        mock_settings.pix2tex_fallback = True
-
-        mock_model = MagicMock(return_value=r"\alpha")
-        mock_page = self._make_fitz_page([plain_text_block])
-        mock_doc = MagicMock()
-        mock_doc.__len__.return_value = 1
-        mock_doc.__getitem__ = MagicMock(return_value=mock_page)
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch("codex.parsing.mathpix._get_pix2tex_model", return_value=mock_model),
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Matrix"),
-            patch("fitz.Rect"),
-        ):
-            from codex.parsing.mathpix import extract_formulas
-
-            results = extract_formulas("paper.pdf")
-
-        assert results == []
-
-    def test_pix2tex_exception_swallowed(self) -> None:
-        """If pix2tex raises, the block is skipped (no exception propagates)."""
-        math_block = (0.0, 0.0, 200.0, 50.0, "∑ α∫β∇γ∂δ∞ε", 0, 0)
-
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = None
-        mock_settings.mathpix_app_key = None
-        mock_settings.pix2tex_fallback = True
-
-        mock_model = MagicMock(side_effect=RuntimeError("GPU OOM"))
-        mock_page = self._make_fitz_page([math_block])
-        mock_doc = MagicMock()
-        mock_doc.__len__.return_value = 1
-        mock_doc.__getitem__ = MagicMock(return_value=mock_page)
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch("codex.parsing.mathpix._get_pix2tex_model", return_value=mock_model),
-            patch("fitz.open", return_value=mock_doc),
-            patch("fitz.Matrix"),
-            patch("fitz.Rect"),
-        ):
-            from PIL import Image
-
-            with patch.object(Image, "frombytes", return_value=MagicMock()):
-                from codex.parsing.mathpix import extract_formulas
-
-                results = extract_formulas("paper.pdf")  # Must not raise
-
-        assert results == []
-
-
-class TestRouteSelection:
-    """Tests for extract_formulas route selection logic."""
-
-    def test_mathpix_selected_when_creds_present(self) -> None:
-        """extract_formulas calls MathPix backend when both creds are set."""
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = SecretStr("my_id")
-        mock_settings.mathpix_app_key = SecretStr("my_key")
-        mock_settings.pix2tex_fallback = True
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch(
-                "codex.parsing.mathpix._extract_formulas_mathpix", return_value=[]
-            ) as mock_mathpix,
-            patch(
-                "codex.parsing.mathpix._extract_formulas_pix2tex", return_value=[]
-            ) as mock_pix2tex,
-        ):
-            from codex.parsing.mathpix import extract_formulas
-
-            extract_formulas("paper.pdf")
-
-        mock_mathpix.assert_called_once_with("paper.pdf", "my_id", "my_key")
-        mock_pix2tex.assert_not_called()
-
-    def test_pix2tex_selected_without_creds(self) -> None:
-        """extract_formulas calls pix2tex when no MathPix creds are set."""
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = None
-        mock_settings.mathpix_app_key = None
-        mock_settings.pix2tex_fallback = True
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch(
-                "codex.parsing.mathpix._extract_formulas_mathpix", return_value=[]
-            ) as mock_mathpix,
-            patch(
-                "codex.parsing.mathpix._extract_formulas_pix2tex", return_value=[]
-            ) as mock_pix2tex,
-        ):
-            from codex.parsing.mathpix import extract_formulas
-
-            extract_formulas("paper.pdf")
-
-        mock_pix2tex.assert_called_once_with("paper.pdf")
-        mock_mathpix.assert_not_called()
-
-    def test_empty_string_creds_treated_as_absent(self) -> None:
-        """Empty string credentials fall through to pix2tex (not MathPix)."""
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = ""
-        mock_settings.mathpix_app_key = ""
-        mock_settings.pix2tex_fallback = True
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch(
-                "codex.parsing.mathpix._extract_formulas_mathpix", return_value=[]
-            ) as mock_mathpix,
-            patch(
-                "codex.parsing.mathpix._extract_formulas_pix2tex", return_value=[]
-            ) as mock_pix2tex,
-        ):
-            from codex.parsing.mathpix import extract_formulas
-
-            extract_formulas("paper.pdf")
-
-        mock_pix2tex.assert_called_once()
-        mock_mathpix.assert_not_called()
-
-    def test_formula_extraction_disabled_when_fallback_false(self) -> None:
-        """No extraction when pix2tex_fallback=False and no creds."""
-        mock_settings = MagicMock()
-        mock_settings.mathpix_app_id = None
-        mock_settings.mathpix_app_key = None
-        mock_settings.pix2tex_fallback = False
-
-        with (
-            patch("codex.parsing.mathpix.get_settings", return_value=mock_settings),
-            patch(
-                "codex.parsing.mathpix._extract_formulas_mathpix", return_value=[]
-            ) as mock_mathpix,
-            patch(
-                "codex.parsing.mathpix._extract_formulas_pix2tex", return_value=[]
-            ) as mock_pix2tex,
-        ):
-            from codex.parsing.mathpix import extract_formulas
-
-            results = extract_formulas("paper.pdf")
-
-        assert results == []
-        mock_mathpix.assert_not_called()
-        mock_pix2tex.assert_not_called()

tests/parsing/test_nougat.py

@@ -1,70 +0,0 @@
-"""Tests for codex.parsing.nougat."""
-
-from __future__ import annotations
-
-from unittest.mock import MagicMock, patch
-
-import httpx
-import pytest
-
-from codex.parsing.nougat import pdf_to_markdown
-
-
-class _FakeResponse:
-    """Minimal stand-in for httpx.Response."""
-
-    def __init__(self, text: str, status_code: int = 200) -> None:
-        self.text = text
-        self.status_code = status_code
-
-    def raise_for_status(self) -> None:
-        if self.status_code >= 400:
-            raise httpx.HTTPStatusError(
-                "error",
-                request=MagicMock(),
-                response=MagicMock(status_code=self.status_code),
-            )
-
-
-class TestPdfToMarkdown:
-    def test_success_returns_response_text(self, tmp_path: pytest.TempdirFactory) -> None:
-        pdf = tmp_path / "paper.pdf"  # type: ignore[operator]
-        pdf.write_bytes(b"%PDF fake")
-
-        expected = "# Title\nContent"
-        fake_response = _FakeResponse(expected)
-
-        with patch("httpx.Client") as mock_client_cls:
-            mock_client = MagicMock()
-            mock_client_cls.return_value.__enter__.return_value = mock_client
-            mock_client.post.return_value = fake_response
-
-            result = pdf_to_markdown(str(pdf), nougat_url="http://localhost:8080")
-
-        assert result == expected
-        mock_client.post.assert_called_once()
-        call_kwargs = mock_client.post.call_args
-        assert "/predict" in call_kwargs[0][0]
-
-    def test_connect_error_retried_exactly_twice(self, tmp_path: pytest.TempdirFactory) -> None:
-        """ConnectError must trigger 2 retries (3 total attempts)."""
-        pdf = tmp_path / "paper.pdf"  # type: ignore[operator]
-        pdf.write_bytes(b"%PDF fake")
-
-        call_count = 0
-
-        def _raising_post(*args: object, **kwargs: object) -> _FakeResponse:
-            nonlocal call_count
-            call_count += 1
-            raise httpx.ConnectError("refused")
-
-        with patch("httpx.Client") as mock_client_cls:
-            mock_client = MagicMock()
-            mock_client_cls.return_value.__enter__.return_value = mock_client
-            mock_client.post.side_effect = _raising_post
-
-            with pytest.raises(httpx.ConnectError):
-                pdf_to_markdown(str(pdf), nougat_url="http://localhost:8080")
-
-        # tenacity: stop_after_attempt(3) → 1 original + 2 retries = 3 calls
-        assert call_count == 3

tests/parsing/test_tex.py

@@ -1,177 +0,0 @@
-"""Tests for codex.parsing.tex."""
-
-from __future__ import annotations
-
-import pytest
-
-from codex.parsing.tex import (
-    chunk_sections,
-    chunk_text,
-    extract_sections,
-    flatten_inputs,
-    latex_to_text,
-)
-
-
-class TestExtractSections:
-    def test_two_sections_returned(self) -> None:
-        latex = r"\section{Intro}" + "\nHello \\cite{foo} world\n"
-        latex += r"\section{Method}" + "\nResult"
-        sections = extract_sections(latex)
-        assert len(sections) == 2
-
-    def test_section_titles(self) -> None:
-        latex = r"\section{Intro}" + "\nHello\n" + r"\section{Method}" + "\nResult"
-        sections = extract_sections(latex)
-        assert sections[0][0] == "Intro"
-        assert sections[1][0] == "Method"
-
-    def test_cite_removed_from_text(self) -> None:
-        latex = r"\section{Intro}" + "\nHello \\cite{foo} world\n"
-        latex += r"\section{Method}" + "\nResult"
-        sections = extract_sections(latex)
-        body = sections[0][1]
-        assert r"\cite" not in body
-        assert "foo" not in body
-
-    def test_no_sections_returns_empty(self) -> None:
-        assert extract_sections("No sections here at all.") == []
-
-    def test_subsection_recognised(self) -> None:
-        latex = r"\subsection{Background}" + "\nSome text."
-        sections = extract_sections(latex)
-        assert len(sections) == 1
-        assert sections[0][0] == "Background"
-
-    def test_comments_stripped(self) -> None:
-        latex = r"\section{A}" + "\nHello % this is a comment\nWorld"
-        sections = extract_sections(latex)
-        assert "%" not in sections[0][1]
-        assert "comment" not in sections[0][1]
-
-    def test_math_stripped(self) -> None:
-        latex = r"\section{A}" + "\nLet $x = 1$ and $$y = 2$$ be numbers."
-        sections = extract_sections(latex)
-        body = sections[0][1]
-        assert "$" not in body
-
-
-class TestChunkText:
-    @pytest.fixture()
-    def long_text(self) -> str:
-        # Build a 600-word string with real sentences
-        sentence = "The quick brown fox jumps over the lazy dog near the river. "
-        words_per_sentence = len(sentence.split())
-        repeats = (600 // words_per_sentence) + 1
-        return (sentence * repeats).strip()
-
-    def test_at_least_two_chunks(self, long_text: str) -> None:
-        chunks = chunk_text(long_text, size=512, overlap=64)
-        assert len(chunks) >= 2
-
-    def test_each_chunk_within_size_bound(self, long_text: str) -> None:
-        size, overlap = 512, 64
-        chunks = chunk_text(long_text, size=size, overlap=overlap)
-        for chunk in chunks:
-            word_count = len(chunk.split())
-            assert word_count <= size + overlap, (
-                f"Chunk has {word_count} words, expected <= {size + overlap}"
-            )
-
-    def test_empty_text_returns_empty_list(self) -> None:
-        assert chunk_text("") == []
-
-    def test_short_text_returns_single_chunk(self) -> None:
-        chunks = chunk_text("Hello world.", size=512, overlap=64)
-        assert len(chunks) == 1
-
-    def test_chunks_cover_all_content(self) -> None:
-        # First chunk must start with first word, last chunk must end with last word
-        text = " ".join(f"word{i}" for i in range(600))
-        chunks = chunk_text(text, size=200, overlap=50)
-        assert chunks[0].startswith("word0")
-        assert chunks[-1].endswith("word599")
-
-
-class TestLatexToText:
-    def test_no_percent_in_output(self) -> None:
-        latex = r"\section{A}" + "\nSome % comment\ntext."
-        result = latex_to_text(latex)
-        assert "%" not in result
-
-    def test_no_cite_in_output(self) -> None:
-        latex = r"\section{A}" + "\nHello \\cite{bar} world."
-        result = latex_to_text(latex)
-        assert r"\cite" not in result
-        assert "bar" not in result
-
-    def test_returns_string(self) -> None:
-        latex = r"\section{A}" + "\nSimple text."
-        result = latex_to_text(latex)
-        assert isinstance(result, str)
-        assert len(result) > 0
-
-    def test_no_section_fallback(self) -> None:
-        latex = r"Some \cite{foo} text with % comment"
-        result = latex_to_text(latex)
-        assert r"\cite" not in result
-        assert "%" not in result
-
-
-class TestFlattenInputs:
-    def test_inlines_input_and_include(self) -> None:
-        files = {
-            "main.tex": r"Start \input{sec/intro} mid \include{proof} end",
-            "sec/intro.tex": "INTRO BODY",
-            "proof.tex": "PROOF BODY",
-        }
-        out = flatten_inputs(files["main.tex"], files)
-        assert "INTRO BODY" in out
-        assert "PROOF BODY" in out
-        assert r"\input" not in out and r"\include" not in out
-
-    def test_resolves_without_tex_suffix(self) -> None:
-        # \input{intro} must resolve the archive member "intro.tex".
-        files = {"main.tex": r"\input{intro}", "intro.tex": "BODY"}
-        assert "BODY" in flatten_inputs(files["main.tex"], files)
-
-    def test_recursive_includes(self) -> None:
-        files = {"a.tex": r"X \input{b}", "b.tex": r"Y \input{c}", "c.tex": "Z"}
-        out = flatten_inputs(files["a.tex"], files)
-        assert "X" in out and "Y" in out and "Z" in out
-
-    def test_unresolved_input_dropped(self) -> None:
-        # A reference with no matching file is removed, not left as a directive.
-        out = flatten_inputs(r"A \input{missing} B", {})
-        assert r"\input" not in out
-        assert "A" in out and "B" in out
-
-    def test_input_cycle_does_not_leak_directive(self) -> None:
-        # a -> b -> a: the depth cap must strip the unresolved directive, not leak it.
-        files = {"a.tex": r"AA \input{b}", "b.tex": r"BB \input{a}"}
-        out = flatten_inputs(files["a.tex"], files)
-        assert r"\input" not in out
-        assert "AA" in out and "BB" in out
-
-    def test_dot_slash_prefix_resolves_without_overstrip(self) -> None:
-        # "./intro" resolves intro.tex; lstrip must not eat a real leading dot/run.
-        files = {"main.tex": r"\input{./intro}", "intro.tex": "BODY"}
-        assert "BODY" in flatten_inputs(files["main.tex"], files)
-
-
-class TestChunkSections:
-    def test_pairs_carry_section_titles(self) -> None:
-        latex = (
-            r"\section{Introduction}" + "\nIntro body text here.\n"
-            r"\section{Proof}" + "\nProof body text here.\n"
-        )
-        pairs = chunk_sections(latex)
-        titles = [t for t, _ in pairs]
-        assert "Introduction" in titles
-        assert "Proof" in titles
-        # No chunk spans a section boundary: each chunk belongs to one title.
-        assert all(isinstance(c, str) and c for _, c in pairs)
-
-    def test_fallback_titleless_when_no_sections(self) -> None:
-        pairs = chunk_sections("Plain text with no section commands at all here.")
-        assert pairs and all(title is None for title, _ in pairs)

tests/provenance/test_provenance.py

@@ -1,209 +0,0 @@
-"""Tests for codex.provenance — @cite scan, code_links CRUD, bib export.
-
-All DB access is mocked; no real PostgreSQL server required.
-"""
-
-from __future__ import annotations
-
-from contextlib import contextmanager
-from datetime import UTC, datetime
-from pathlib import Path
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-from codex.models import CodeLink
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _make_conn_cm(mock_conn: MagicMock) -> Any:
-    """Return a context-manager factory that yields *mock_conn*."""
-
-    @contextmanager  # type: ignore[arg-type]
-    def _cm() -> Any:
-        yield mock_conn
-
-    return _cm
-
-
-def _dict_row(**kwargs: Any) -> dict[str, Any]:
-    return dict(kwargs)
-
-
-# ---------------------------------------------------------------------------
-# 1. test_scan_cite_tags
-# ---------------------------------------------------------------------------
-
-
-def test_scan_cite_tags(tmp_path: Path) -> None:
-    """A .cpp file with @cite produces correct dict entries."""
-    from codex.provenance import scan_cite_tags
-
-    src_file = tmp_path / "algo.cpp"
-    src_file.write_text(
-        "// Helper function\n"
-        "void compute_hodge() {\n"
-        "    // @cite somepaper2023\n"
-        "    do_stuff();\n"
-        "}\n",
-        encoding="utf-8",
-    )
-
-    results = scan_cite_tags(str(tmp_path))
-
-    assert len(results) == 1
-    hit = results[0]
-    assert hit["bibkey"] == "somepaper2023"
-    assert hit["line"] == "3"
-    assert hit["file"] == "algo.cpp"
-    # symbol extracted from "compute_hodge()" on the line above
-    assert hit["symbol"] == "compute_hodge"
-
-
-# ---------------------------------------------------------------------------
-# 2. test_scan_cite_tags_no_files
-# ---------------------------------------------------------------------------
-
-
-def test_scan_cite_tags_no_files(tmp_path: Path) -> None:
-    """Empty directory returns an empty list."""
-    from codex.provenance import scan_cite_tags
-
-    result = scan_cite_tags(str(tmp_path))
-    assert result == []
-
-
-# ---------------------------------------------------------------------------
-# 3. test_add_code_link
-# ---------------------------------------------------------------------------
-
-
-def test_add_code_link() -> None:
-    """Mock conn, verify INSERT is issued and returned CodeLink has id from RETURNING."""
-    from codex.provenance import add_code_link
-
-    now = datetime(2026, 6, 5, tzinfo=UTC)
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchone.return_value = _dict_row(id=42, added_at=now)
-
-    with patch("codex.provenance.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        link = add_code_link(
-            symbol="dec::hodge_star",
-            paper_id="2301.07041",
-            role="implements Thm 3.2",
-            note="core algorithm",
-        )
-
-    assert isinstance(link, CodeLink)
-    assert link.id == 42
-    assert link.added_at == now
-    assert link.symbol == "dec::hodge_star"
-    assert link.paper_id == "2301.07041"
-    assert link.role == "implements Thm 3.2"
-    assert link.note == "core algorithm"
-
-    call_args = mock_conn.execute.call_args
-    sql: str = call_args[0][0]
-    assert "INSERT INTO code_links" in sql
-    assert "RETURNING" in sql
-
-    mock_conn.commit.assert_called_once()
-
-
-# ---------------------------------------------------------------------------
-# 4. test_list_code_links_all
-# ---------------------------------------------------------------------------
-
-
-def test_list_code_links_all() -> None:
-    """No filter → all rows returned as CodeLink list."""
-    from codex.provenance import list_code_links
-
-    now = datetime(2026, 6, 5, tzinfo=UTC)
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _dict_row(id=1, symbol="foo::bar", paper_id="W111", role="uses", note=None, added_at=now),
-        _dict_row(id=2, symbol="baz::qux", paper_id=None, role=None, note="ref", added_at=now),
-    ]
-
-    with patch("codex.provenance.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        links = list_code_links()
-
-    assert len(links) == 2
-    assert all(isinstance(lnk, CodeLink) for lnk in links)
-    assert links[0].symbol == "foo::bar"
-    assert links[0].id == 1
-    assert links[1].symbol == "baz::qux"
-    assert links[1].paper_id is None
-
-    # No WHERE clause in the SQL
-    call_args = mock_conn.execute.call_args
-    sql: str = call_args[0][0]
-    assert "WHERE" not in sql
-
-
-# ---------------------------------------------------------------------------
-# 5. test_list_code_links_filtered
-# ---------------------------------------------------------------------------
-
-
-def test_list_code_links_filtered() -> None:
-    """Filter by paper_id → WHERE clause applied and only matching rows returned."""
-    from codex.provenance import list_code_links
-
-    now = datetime(2026, 6, 5, tzinfo=UTC)
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _dict_row(
-            id=3,
-            symbol="some::func",
-            paper_id="2301.07041",
-            role=None,
-            note=None,
-            added_at=now,
-        ),
-    ]
-
-    with patch("codex.provenance.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        links = list_code_links(paper_id="2301.07041")
-
-    assert len(links) == 1
-    assert links[0].paper_id == "2301.07041"
-
-    call_args = mock_conn.execute.call_args
-    sql: str = call_args[0][0]
-    params: dict[str, Any] = call_args[0][1]
-    assert "WHERE paper_id" in sql
-    assert params["paper_id"] == "2301.07041"
-
-
-# ---------------------------------------------------------------------------
-# 6. test_export_bib_format
-# ---------------------------------------------------------------------------
-
-
-def test_export_bib_format() -> None:
-    """Mock returns one paper row; verify valid BibTeX string output."""
-    from codex.provenance import export_bib
-
-    mock_conn = MagicMock()
-    mock_conn.execute.return_value.fetchall.return_value = [
-        _dict_row(
-            id="2301.07041",
-            bibkey="smith2023conformal",
-            title="Conformal Approaches to Laplacian Eigenmaps",
-            authors=["Alice Smith", "Bob Jones"],
-            year=2023,
-        ),
-    ]
-
-    with patch("codex.provenance.get_conn", side_effect=_make_conn_cm(mock_conn)):
-        bib = export_bib()
-
-    assert "@article{smith2023conformal," in bib
-    assert "title  = {Conformal Approaches to Laplacian Eigenmaps}" in bib
-    assert "author = {Alice Smith and Bob Jones}" in bib
-    assert "year   = {2023}" in bib
-    assert "note   = {2301.07041}" in bib

tests/quality/test_quality.py

@@ -1,302 +0,0 @@
-"""Tests for codex.quality — chunk quality gate and section classification (F-16)."""
-
-from __future__ import annotations
-
-from unittest.mock import MagicMock
-
-from codex.quality import (
-    _bib_score,
-    _clean_title,
-    classify_section,
-    filter_chunks,
-    is_quality_chunk,
-    run_quality_pass,
-    section_label,
-)
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _settings(min_chars: int = 60, min_alpha: float = 0.40, max_bib: float = 0.70):
-    s = MagicMock()
-    s.chunk_min_chars = min_chars
-    s.chunk_min_alpha_ratio = min_alpha
-    s.chunk_max_bib_score = max_bib
-    return s
-
-
-GOOD_TEXT = (
-    "In differential geometry, the discrete Laplace–Beltrami operator "
-    "generalises the smooth operator to polyhedral surfaces by using cotangent "
-    "weights. This discretisation preserves key spectral properties."
-)
-
-BIB_TEXT = (
-    "Bobenko, A. and Pinkall, U. (2015) Discrete differential geometry. "
-    "doi:10.1145/2010324.1964978 Smith et al. (2018) Surface remeshing. "
-    "doi:10.1007/s00211-019-01082-w Jones et al. (2020) doi:10.1145/1234567.890123"
-)
-
-
-# ---------------------------------------------------------------------------
-# _bib_score
-# ---------------------------------------------------------------------------
-
-
-class TestBibScore:
-    def test_empty_string(self):
-        assert _bib_score("") == 0.0
-
-    def test_clean_prose_is_low(self):
-        assert _bib_score(GOOD_TEXT) < 0.10
-
-    def test_bib_like_text_is_high(self):
-        assert _bib_score(BIB_TEXT) > 0.70
-
-    def test_single_doi_short_text_spikes(self):
-        text = "See doi:10.1145/2010324.1964978 for details."
-        # 8 words, 1 DOI (weight 3) → raw = 3 / max(0.8, 1) = 3.0 → capped 1.0
-        assert _bib_score(text) == 1.0
-
-    def test_score_capped_at_one(self):
-        # _DOI_RE requires \d{4,} — use a valid registrant length
-        heavy = "10.1145/abcdef " * 20
-        assert _bib_score(heavy) == 1.0
-
-
-# ---------------------------------------------------------------------------
-# is_quality_chunk
-# ---------------------------------------------------------------------------
-
-
-class TestIsQualityChunk:
-    def test_passes_good_text(self):
-        assert is_quality_chunk(GOOD_TEXT, settings=_settings()) is True
-
-    def test_fails_too_short(self):
-        assert is_quality_chunk("Short.", settings=_settings(min_chars=60)) is False
-
-    def test_fails_low_alpha_ratio(self):
-        # Mostly digits and symbols
-        junk = "1234 5678 9012 3456 7890 |||| ==== ---- ____ .... ;;;; 0000 9999 8888 7777"
-        assert is_quality_chunk(junk, settings=_settings(min_alpha=0.40)) is False
-
-    def test_fails_high_bib_score(self):
-        assert is_quality_chunk(BIB_TEXT, settings=_settings(max_bib=0.70)) is False
-
-    def test_bib_threshold_is_strict_gt(self):
-        # A text that scores exactly at the threshold should still pass (> not >=)
-        s = _settings(max_bib=1.0)
-        assert is_quality_chunk(GOOD_TEXT, settings=s) is True
-
-    def test_exactly_min_chars_passes(self):
-        # check is `len < min_chars`, so exactly at the threshold passes
-        text = "a" * 60
-        assert is_quality_chunk(text, settings=_settings(min_chars=60)) is True
-
-    def test_one_below_min_chars_fails(self):
-        text = "a" * 59
-        assert is_quality_chunk(text, settings=_settings(min_chars=60)) is False
-
-
-# ---------------------------------------------------------------------------
-# classify_section
-# ---------------------------------------------------------------------------
-
-
-class TestClassifySection:
-    def test_abstract_header(self):
-        assert classify_section("Abstract\n\nThis paper presents...") == "abstract"
-
-    def test_abstract_german(self):
-        assert classify_section("Zusammenfassung\nDiese Arbeit...") == "abstract"
-
-    def test_intro_header(self):
-        assert classify_section("Introduction\nOver the past decade...") == "intro"
-
-    def test_intro_numbered(self):
-        assert classify_section("1. Introduction\nWe study...") == "intro"
-
-    def test_theorem_header(self):
-        assert classify_section("Theorem 3.1 (Existence). Let M be...") == "theorem"
-
-    def test_lemma_header(self):
-        assert classify_section("Lemma 2. If f is smooth then...") == "theorem"
-
-    def test_proof_header(self):
-        assert classify_section("Proof. We proceed by induction...") == "proof"
-
-    def test_bibliography_header(self):
-        assert classify_section("References\n[1] Bobenko, A....") == "bibliography"
-
-    def test_bibliography_german(self):
-        assert classify_section("Literatur\n[1] Müller, H....") == "bibliography"
-
-    def test_body_fallback(self):
-        assert classify_section("The cotangent weight for edge (i,j) is...") == "body"
-
-    def test_doi_density_fallback_to_bibliography(self):
-        # No "References" header but heavy DOI density → bibliography
-        dense = BIB_TEXT
-        assert classify_section(dense) == "bibliography"
-
-    def test_only_first_200_chars_checked(self):
-        # Pattern in position >200 should NOT be matched as section header
-        prefix = "x" * 201
-        text = prefix + "\nAbstract"
-        assert classify_section(text) == "body"
-
-
-# ---------------------------------------------------------------------------
-# filter_chunks
-# ---------------------------------------------------------------------------
-
-
-class TestFilterChunks:
-    def test_empty_list(self):
-        assert filter_chunks([], settings=_settings()) == []
-
-    def test_all_good(self):
-        chunks = [GOOD_TEXT, GOOD_TEXT + " More content here."]
-        assert filter_chunks(chunks, settings=_settings()) == chunks
-
-    def test_all_bad(self):
-        chunks = ["x", "y"]
-        assert filter_chunks(chunks, settings=_settings()) == []
-
-    def test_mixed(self):
-        chunks = [GOOD_TEXT, "Short.", GOOD_TEXT]
-        result = filter_chunks(chunks, settings=_settings())
-        assert len(result) == 2
-        assert all(c == GOOD_TEXT for c in result)
-
-
-# ---------------------------------------------------------------------------
-# run_quality_pass
-# ---------------------------------------------------------------------------
-
-
-def _make_mock_conn(rows: list[dict]) -> MagicMock:
-    """Return a mock psycopg connection whose execute().fetchall() returns rows."""
-    conn = MagicMock()
-    conn.execute.return_value.fetchall.return_value = rows
-    return conn
-
-
-class TestRunQualityPass:
-    def test_removes_bad_chunk(self):
-        conn = _make_mock_conn([{"id": 1, "content": "x"}])
-        stats = run_quality_pass(conn=conn, settings=_settings())
-        assert stats["removed"] == 1
-        assert stats["kept"] == 0
-        # The DELETE must have been called with chunk id=1
-        delete_calls = [c for c in conn.execute.call_args_list if "DELETE" in str(c)]
-        assert len(delete_calls) == 1
-
-    def test_tags_good_chunk(self):
-        conn = _make_mock_conn([{"id": 2, "content": GOOD_TEXT}])
-        stats = run_quality_pass(conn=conn, settings=_settings())
-        assert stats["kept"] == 1
-        assert stats["tagged"] == 1
-        assert stats["removed"] == 0
-        # UPDATE must have been called
-        update_calls = [c for c in conn.execute.call_args_list if "UPDATE" in str(c)]
-        assert len(update_calls) == 1
-
-    def test_mixed_batch(self):
-        rows = [
-            {"id": 1, "content": "x"},
-            {"id": 2, "content": GOOD_TEXT},
-            {"id": 3, "content": "y"},
-            {"id": 4, "content": GOOD_TEXT + " extra words here."},
-        ]
-        conn = _make_mock_conn(rows)
-        stats = run_quality_pass(conn=conn, settings=_settings())
-        assert stats["removed"] == 2
-        assert stats["kept"] == 2
-        assert stats["tagged"] == 2
-
-    def test_paper_id_scopes_query(self):
-        conn = _make_mock_conn([])
-        run_quality_pass(paper_id="2301.07041", conn=conn, settings=_settings())
-        first_call = conn.execute.call_args_list[0]
-        sql = first_call[0][0]
-        assert "paper_id" in sql
-
-    def test_all_papers_query_has_no_filter(self):
-        conn = _make_mock_conn([])
-        run_quality_pass(paper_id=None, conn=conn, settings=_settings())
-        first_call = conn.execute.call_args_list[0]
-        sql = first_call[0][0]
-        assert "paper_id" not in sql
-
-    def test_commit_is_called(self):
-        conn = _make_mock_conn([])
-        run_quality_pass(conn=conn, settings=_settings())
-        conn.commit.assert_called_once()
-
-    def test_section_label_stored_correctly(self):
-        conn = _make_mock_conn([{"id": 5, "content": "Abstract\n\n" + GOOD_TEXT}])
-        run_quality_pass(conn=conn, settings=_settings())
-        update_calls = [c for c in conn.execute.call_args_list if "UPDATE" in str(c)]
-        assert len(update_calls) == 1
-        kwargs = update_calls[0][0][1]
-        assert kwargs["section"] == "abstract"
-        assert kwargs["id"] == 5
-
-
-class TestCleanTitle:
-    def test_strips_leading_numbering(self):
-        assert _clean_title("3.2 Variational Principles") == "variational principles"
-
-    def test_strips_latex_and_braces(self):
-        assert _clean_title(r"The \emph{Discrete} Laplacian") == "the discrete laplacian"
-
-    def test_collapses_and_lowercases(self):
-        assert _clean_title("  Rigidity   Results  ") == "rigidity results"
-
-    def test_strips_accent_macros(self):
-        assert _clean_title(r"M\"obius Invariance") == "mobius invariance"
-
-    def test_strips_ties_and_accents(self):
-        assert _clean_title(r"Colin~de~Verdi\`ere") == "colin de verdiere"
-
-    def test_truncates_at_word_boundary(self):
-        long = "alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu omega"
-        out = _clean_title(long)
-        assert len(out) <= 60
-        assert not out.endswith(" ")
-        assert long.lower().startswith(out)  # whole words from the start, no partial tail
-
-
-class TestSectionLabel:
-    def test_bare_canonical_heading_maps_to_bucket(self):
-        # Only an EXACT canonical heading collapses to its controlled bucket.
-        assert section_label("Introduction", "body text") == "intro"
-        assert section_label("Proof", "body text") == "proof"
-        assert section_label("References", "body text") == "bibliography"
-        assert section_label("Abstract", "body text") == "abstract"
-
-    def test_descriptive_title_stored_verbatim(self):
-        # R-F: a heading that merely STARTS with a keyword keeps its real title,
-        # not the bucket — fixes classify_section prefix-match over-bucketing.
-        assert section_label("Preliminaries", "body text") == "preliminaries"
-        assert section_label("3.1 Discrete Conformal Maps", "x") == "discrete conformal maps"
-        assert section_label("Proof of Theorem 3", "x") == "proof of theorem 3"
-        assert section_label("Abstract Nonsense and Categories", "x") == (
-            "abstract nonsense and categories"
-        )
-        assert section_label("Introduction to Operator Algebras", "x") == (
-            "introduction to operator algebras"
-        )
-
-    def test_none_or_blank_title_falls_back_to_content(self):
-        # .pdf/.txt path: unchanged content-based classification.
-        assert section_label(None, "Theorem 1. Let X be ...") == "theorem"
-        assert section_label("", "ordinary body prose here") == "body"
-
-    def test_title_cleaning_to_empty_falls_back_to_content(self):
-        # A pure-numbering heading cleans to "" → classify by content instead.
-        assert section_label("3.2", "Lemma 2. Suppose ...") == "theorem"

tests/scaffold/test_config.py

@@ -1,38 +0,0 @@
-"""Smoke tests for codex.config.Settings."""
-
-from __future__ import annotations
-
-from codex.config import Settings, get_settings
-
-
-def test_settings_defaults() -> None:
-    """Settings() is constructable without env vars; expected defaults are set."""
-    s = Settings()
-    assert (
-        s.database_url.get_secret_value()
-        == "postgresql://researcher:change_me@localhost:5432/papers"
-    )
-    assert s.embedding_model == "BAAI/bge-m3"
-    assert s.embedding_dim == 1024
-
-
-def test_settings_env_override(monkeypatch: object) -> None:
-    """DATABASE_URL env var overrides the default database_url."""
-    import pytest
-
-    mp: pytest.MonkeyPatch = monkeypatch  # type: ignore[assignment]
-    mp.setenv("DATABASE_URL", "postgresql://x:y@h:5432/db")
-    s = Settings()
-    assert s.database_url.get_secret_value() == "postgresql://x:y@h:5432/db"
-
-
-def test_get_settings_is_singleton() -> None:
-    """get_settings() returns the same object on repeated calls; new object after cache_clear."""
-    get_settings.cache_clear()
-    first = get_settings()
-    second = get_settings()
-    assert first is second
-
-    get_settings.cache_clear()
-    third = get_settings()
-    assert third is not first

tests/scaffold/test_db.py

@@ -1,72 +0,0 @@
-"""Smoke tests for codex.db — get_conn and apply_schema."""
-
-from __future__ import annotations
-
-from unittest.mock import MagicMock, patch
-
-
-def test_get_conn_context_manager() -> None:
-    """get_conn() yields a context manager that calls psycopg.connect and cleans up."""
-    from codex.db import get_conn
-
-    mock_conn = MagicMock()
-    mock_connect = MagicMock()
-    # psycopg.connect is itself used as a context manager inside get_conn
-    mock_connect.return_value.__enter__ = MagicMock(return_value=mock_conn)
-    mock_connect.return_value.__exit__ = MagicMock(return_value=False)
-
-    with patch("codex.db.psycopg.connect", mock_connect), get_conn() as conn:
-        assert conn is mock_conn
-
-    mock_connect.assert_called_once()
-
-
-def test_apply_schema_executes_sql() -> None:
-    """apply_schema(conn) calls conn.execute and conn.commit."""
-    from codex.db import apply_schema
-
-    mock_conn = MagicMock()
-
-    # Provide a fake schema.sql so Path.read_text succeeds
-    with patch("codex.db.Path.read_text", return_value="CREATE TABLE IF NOT EXISTS test ();"):
-        apply_schema(mock_conn)
-
-    mock_conn.execute.assert_called_once()
-    mock_conn.commit.assert_called_once()
-
-
-def test_schema_sql_is_idempotent() -> None:
-    """Every CREATE TABLE/INDEX in infra/schema.sql uses IF NOT EXISTS (audit M-1, T-2).
-
-    Re-applying schema.sql must be a safe no-op so the migration can run
-    repeatedly against an existing database.
-    """
-    import re
-    from pathlib import Path
-
-    schema = (Path(__file__).resolve().parents[2] / "infra" / "schema.sql").read_text()
-    # Drop line comments so commented-out example queries don't count.
-    code = "\n".join(line.split("--", 1)[0] for line in schema.splitlines())
-    non_idempotent = re.findall(
-        r"CREATE\s+(?:UNIQUE\s+)?(?:TABLE|INDEX)\s+(?!IF\s+NOT\s+EXISTS)(\w+)",
-        code,
-        re.IGNORECASE,
-    )
-    assert not non_idempotent, f"schema.sql has non-idempotent CREATE statements: {non_idempotent}"
-
-
-def test_migrate_command_reports_privilege_error() -> None:
-    """`codex migrate` surfaces InsufficientPrivilege with guidance and exits 1 (M-1)."""
-    import psycopg
-    from typer.testing import CliRunner
-
-    from codex.cli import app
-
-    def boom(*args: object, **kwargs: object) -> object:
-        raise psycopg.errors.InsufficientPrivilege("must be owner of table chunks")
-
-    with patch("psycopg.connect", boom):
-        result = CliRunner().invoke(app, ["migrate"])
-
-    assert result.exit_code == 1
-    assert "MIGRATION_DATABASE_URL" in result.output

tests/scaffold/test_models.py

@@ -1,34 +0,0 @@
-"""Smoke tests for codex.models dataclasses."""
-
-from __future__ import annotations
-
-from codex.models import Chunk, Citation, CodeLink, Paper
-
-
-def test_paper_defaults() -> None:
-    """Paper(id=..., title=...) has correct optional-field defaults."""
-    p = Paper(id="2301.00001", title="T")
-    assert p.openalex_id is None
-    assert p.authors == []
-    assert p.year is None
-
-
-def test_chunk_required_fields() -> None:
-    """Chunk(paper_id=..., ord=..., content=...) has id=None and embedding=None."""
-    c = Chunk(paper_id="x", ord=0, content="c")
-    assert c.id is None
-    assert c.embedding is None
-
-
-def test_citation_fields() -> None:
-    """Citation(citing_id=..., cited_id=...) has context=None."""
-    cit = Citation(citing_id="a", cited_id="b")
-    assert cit.context is None
-
-
-def test_codelink_fields() -> None:
-    """CodeLink(symbol=...) has paper_id=None, role=None, and id=None."""
-    cl = CodeLink(symbol="foo")
-    assert cl.paper_id is None
-    assert cl.role is None
-    assert cl.id is None

tests/scripts/test_ra_grobid_backfill.py

@@ -1,112 +0,0 @@
-"""Tests for scripts.ra_grobid_backfill — the R-A GROBID references backfill.
-
-Only the pure, offline logic is covered (id normalization + Citation assembly);
-``extract_references`` is mocked so no GROBID server or DB is required.
-"""
-
-from __future__ import annotations
-
-from contextlib import contextmanager
-from typing import Any
-from unittest.mock import MagicMock, patch
-
-from codex.models import Citation
-from scripts.ra_grobid_backfill import _normalize_cited_id, build_grobid_citations, main
-
-# ---------------------------------------------------------------------------
-# _normalize_cited_id
-# ---------------------------------------------------------------------------
-
-
-def test_normalize_doi_url_to_bare_lowercase() -> None:
-    assert _normalize_cited_id(doi="https://doi.org/10.1007/S00454-019-00132-8") == (
-        "10.1007/s00454-019-00132-8"
-    )
-
-
-def test_normalize_doi_prefixes() -> None:
-    assert _normalize_cited_id(doi="http://doi.org/10.1/X") == "10.1/x"
-    assert _normalize_cited_id(doi="doi:10.1/X") == "10.1/x"
-    assert _normalize_cited_id(doi="10.1/x") == "10.1/x"
-
-
-def test_normalize_arxiv_prefix_stripped() -> None:
-    assert _normalize_cited_id(arxiv_id="arXiv:2305.10988") == "2305.10988"
-    assert _normalize_cited_id(arxiv_id="2305.10988") == "2305.10988"
-    # Legacy ids keep their slash.
-    assert _normalize_cited_id(arxiv_id="math/0603097") == "math/0603097"
-
-
-def test_normalize_doi_takes_precedence_over_arxiv() -> None:
-    assert _normalize_cited_id(doi="10.5/y", arxiv_id="2305.10988") == "10.5/y"
-
-
-def test_normalize_empty_returns_none() -> None:
-    assert _normalize_cited_id() is None
-    assert _normalize_cited_id(doi="   ", arxiv_id="") is None
-
-
-# ---------------------------------------------------------------------------
-# build_grobid_citations
-# ---------------------------------------------------------------------------
-
-
-def _ref(doi: str = "", arxiv_id: str = "") -> dict[str, str]:
-    return {"title": "t", "authors": "a", "year": "2020", "doi": doi, "arxiv_id": arxiv_id}
-
-
-def test_build_citations_normalizes_dedups_and_drops_self() -> None:
-    paper_id = "10.1007/s00454-019-00132-8"
-    refs = [
-        _ref(doi="https://doi.org/10.1/A"),  # → 10.1/a
-        _ref(doi="DOI:10.1/a"),  # duplicate of the above after normalization
-        _ref(arxiv_id="arXiv:2305.10988"),  # → 2305.10988
-        _ref(doi="", arxiv_id=""),  # no id → skipped
-        _ref(doi="https://doi.org/10.1007/S00454-019-00132-8"),  # self-cite → dropped
-    ]
-    with patch("scripts.ra_grobid_backfill.extract_references", return_value=refs) as mock_ex:
-        cits = build_grobid_citations(paper_id, "some.pdf", grobid_url="http://grobid")
-
-    mock_ex.assert_called_once_with("some.pdf", grobid_url="http://grobid", timeout=60.0)
-    assert all(isinstance(c, Citation) for c in cits)
-    assert all(c.citing_id == paper_id for c in cits)
-    assert [c.cited_id for c in cits] == ["10.1/a", "2305.10988"]
-
-
-def test_build_citations_empty_when_no_refs() -> None:
-    with patch("scripts.ra_grobid_backfill.extract_references", return_value=[]):
-        assert build_grobid_citations("10.1/x", "x.pdf") == []
-
-
-# ---------------------------------------------------------------------------
-# main — connection scope (review finding #4)
-# ---------------------------------------------------------------------------
-
-
-def test_main_uses_separate_connections_for_read_and_write(tmp_path: Any) -> None:
-    """Review #4: the read connection is released BEFORE the slow GROBID loop, and a
-    fresh connection is opened for the write — so no idle-in-transaction connection
-    spans the per-paper network loop (which would abort on an SSH-tunnel drop)."""
-    (tmp_path / "paper.pdf").write_bytes(b"%PDF-1.4")
-    conns: list[MagicMock] = []
-
-    @contextmanager
-    def fake_get_conn() -> Any:
-        c = MagicMock()
-        c.execute.return_value.fetchall.return_value = [
-            {"id": "10.1/x", "source_path": "/d/paper.txt", "out_edges": 0}
-        ]
-        c.execute.return_value.fetchone.return_value = {"papers": 1, "citing": 1, "edges": 1}
-        conns.append(c)
-        yield c
-
-    with (
-        patch("scripts.ra_grobid_backfill.get_conn", side_effect=fake_get_conn),
-        patch("scripts.ra_grobid_backfill.build_grobid_citations", return_value=[]),
-    ):
-        rc = main(["--pdf-dir", str(tmp_path), "--write"])
-
-    assert rc == 0
-    # Two DISTINCT get_conn() context managers (read, then write) — not one held
-    # open across the GROBID loop.
-    assert len(conns) == 2

tests/scripts/test_rc_tex_reingest.py

@@ -1,21 +0,0 @@
-"""Tests for scripts.rc_tex_reingest."""
-
-from __future__ import annotations
-
-from scripts.rc_tex_reingest import is_arxiv_id
-
-
-def test_is_arxiv_id_accepts_modern_and_legacy() -> None:
-    assert is_arxiv_id("2305.10988")
-    assert is_arxiv_id("math/0603097")
-    assert is_arxiv_id("1005.2698")
-
-
-def test_is_arxiv_id_rejects_dois_and_wids() -> None:
-    assert not is_arxiv_id("10.1007/s00454-019-00132-8")
-    assert not is_arxiv_id("10.14279/depositonce-20357")
-    assert not is_arxiv_id("W2971636899")
-    assert not is_arxiv_id("https://openalex.org/W2971636899")
-    assert not is_arxiv_id("http://openalex.org/W2971636899")
-    assert not is_arxiv_id("10.48550/arXiv.2305.10988")  # arXiv DOI form, not bare id
-    assert not is_arxiv_id("")

tests/sources/test_arxiv.py

@@ -1,288 +0,0 @@
-"""Tests for codex.sources.arxiv."""
-
-from __future__ import annotations
-
-import io
-import tarfile
-
-import httpx
-import pytest
-
-from codex.sources import arxiv
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _make_tar_gz(files: dict[str, bytes]) -> bytes:
-    """Build an in-memory .tar.gz archive from a dict of {name: content}."""
-    buf = io.BytesIO()
-    with tarfile.open(fileobj=buf, mode="w:gz") as tf:
-        for name, content in files.items():
-            info = tarfile.TarInfo(name=name)
-            info.size = len(content)
-            tf.addfile(info, io.BytesIO(content))
-    return buf.getvalue()
-
-
-# ---------------------------------------------------------------------------
-# fetch_source — success: in-memory tar.gz with one .tex
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_source_returns_latex(monkeypatch: pytest.MonkeyPatch) -> None:
-    """fetch_source extracts and returns the LaTeX content from a .tar.gz."""
-    latex_content = b"\\documentclass{article}\n\\begin{document}\nHello world\n\\end{document}"
-    tar_bytes = _make_tar_gz({"main.tex": latex_content})
-
-    def mock_get(
-        url: str,
-        *,
-        timeout: int = 60,
-        follow_redirects: bool = True,
-    ) -> httpx.Response:
-        return httpx.Response(200, content=tar_bytes)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-
-    result = arxiv.fetch_source("2301.07041")
-
-    assert result is not None
-    assert "\\documentclass" in result
-    assert "Hello world" in result
-
-
-def test_fetch_source_flattens_multifile(monkeypatch: pytest.MonkeyPatch) -> None:
-    """Multi-file projects: \\input/\\include directives are inlined so the body
-    from the included files is returned, not just the primary skeleton (R-C)."""
-    main = (
-        b"\\documentclass{article}\\begin{document}"
-        b"\\input{sections/intro}\\include{proof}\\end{document}"
-    )
-    tar_bytes = _make_tar_gz(
-        {
-            "main.tex": main,
-            "sections/intro.tex": b"INTRODUCTION BODY TEXT",
-            "proof.tex": b"PROOF BODY TEXT",
-        }
-    )
-
-    def mock_get(
-        url: str,
-        *,
-        timeout: int = 60,
-        follow_redirects: bool = True,
-    ) -> httpx.Response:
-        return httpx.Response(200, content=tar_bytes)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-
-    result = arxiv.fetch_source("2301.07041")
-
-    assert result is not None
-    assert "INTRODUCTION BODY TEXT" in result  # \input was inlined
-    assert "PROOF BODY TEXT" in result  # \include was inlined
-    assert "\\input" not in result and "\\include" not in result
-
-
-def test_fetch_source_bare_gzip_single_file(monkeypatch: pytest.MonkeyPatch) -> None:
-    """Legacy arXiv submissions are served as a single bare gzipped .tex (no tar
-    wrapper); fetch_source must gunzip the body directly rather than fail."""
-    import gzip
-
-    latex = b"\\documentstyle[a4]{article}\n\\section{Intro}\nOld single-file paper body."
-    gz_bytes = gzip.compress(latex)
-
-    def mock_get(
-        url: str,
-        *,
-        timeout: int = 60,
-        follow_redirects: bool = True,
-    ) -> httpx.Response:
-        return httpx.Response(200, content=gz_bytes)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-
-    result = arxiv.fetch_source("math/0001176")
-
-    assert result is not None
-    assert "Old single-file paper body." in result
-
-
-def test_fetch_source_resolves_non_tex_include(monkeypatch: pytest.MonkeyPatch) -> None:
-    """\\input of a non-.tex body member (e.g. a .def file) is inlined, not dropped (R-C)."""
-    main = b"\\documentclass{article}\\begin{document}\\input{body.def}\\end{document}"
-    tar_bytes = _make_tar_gz({"main.tex": main, "body.def": b"NON TEX BODY TEXT"})
-
-    def mock_get(url: str, *, timeout: int = 60, follow_redirects: bool = True) -> httpx.Response:
-        return httpx.Response(200, content=tar_bytes)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-    result = arxiv.fetch_source("2301.07041")
-    assert result is not None
-    assert "NON TEX BODY TEXT" in result
-
-
-def test_fetch_source_ignores_commented_documentclass(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A commented %\\documentclass must not select a file as the primary document."""
-    decoy = b"% \\documentclass{article}\n\\section{Stub}\nDECOY ONLY"
-    real = b"\\documentclass{book}\\begin{document}REAL BODY HERE\\end{document}"
-    # decoy sorts/inserts first; without the fix it would win on the substring match.
-    tar_bytes = _make_tar_gz({"aaa_decoy.tex": decoy, "real.tex": real})
-
-    def mock_get(url: str, *, timeout: int = 60, follow_redirects: bool = True) -> httpx.Response:
-        return httpx.Response(200, content=tar_bytes)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-    result = arxiv.fetch_source("2301.07041")
-    assert result is not None
-    assert "REAL BODY HERE" in result
-    assert "DECOY ONLY" not in result
-
-
-# ---------------------------------------------------------------------------
-# fetch_source — 404 → None
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_source_404_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A 404 response should return None."""
-
-    def mock_get(
-        url: str,
-        *,
-        timeout: int = 60,
-        follow_redirects: bool = True,
-    ) -> httpx.Response:
-        return httpx.Response(404)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-
-    result = arxiv.fetch_source("nonexistent-id")
-    assert result is None
-
-
-# ---------------------------------------------------------------------------
-# fetch_source — no .tex in archive → None
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_source_no_tex_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    """If the archive contains no .tex files, return None."""
-    tar_bytes = _make_tar_gz({"README.md": b"# Paper\nNo LaTeX here."})
-
-    def mock_get(
-        url: str,
-        *,
-        timeout: int = 60,
-        follow_redirects: bool = True,
-    ) -> httpx.Response:
-        return httpx.Response(200, content=tar_bytes)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-
-    result = arxiv.fetch_source("2301.07041")
-    assert result is None
-
-
-# ---------------------------------------------------------------------------
-# fetch_source — fallback to largest .tex when no \documentclass
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_source_fallback_to_largest_tex(monkeypatch: pytest.MonkeyPatch) -> None:
-    """When no file has \\documentclass, the largest .tex is returned."""
-    small_tex = b"% small file\n\\section{Intro}"
-    large_tex = b"% large file\n" + b"x" * 500
-
-    tar_bytes = _make_tar_gz({"small.tex": small_tex, "large.tex": large_tex})
-
-    def mock_get(
-        url: str,
-        *,
-        timeout: int = 60,
-        follow_redirects: bool = True,
-    ) -> httpx.Response:
-        return httpx.Response(200, content=tar_bytes)
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-
-    result = arxiv.fetch_source("2301.07041")
-
-    assert result is not None
-    assert "large file" in result
-
-
-# ---------------------------------------------------------------------------
-# fetch_pdf_url — pure computation, no mock needed
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_pdf_url_pure_computation() -> None:
-    """fetch_pdf_url returns the correct URL without making any HTTP request."""
-    url = arxiv.fetch_pdf_url("2301.07041")
-    assert url == "https://arxiv.org/pdf/2301.07041.pdf"
-
-    url2 = arxiv.fetch_pdf_url("1234.56789")
-    assert url2 == "https://arxiv.org/pdf/1234.56789.pdf"
-
-
-# ---------------------------------------------------------------------------
-# fetch_metadata — parse the Atom feed into a Paper (DQ-2)
-# ---------------------------------------------------------------------------
-
-_ATOM_FEED = """<?xml version="1.0" encoding="UTF-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom">
-  <entry>
-    <title>A discrete version of Liouville's theorem on conformal maps</title>
-    <summary>Liouville's theorem says that in dimension greater than two,
-    all conformal maps are Moebius transformations.</summary>
-    <published>2019-11-03T00:00:00Z</published>
-    <author><name>Ulrich Pinkall</name></author>
-    <author><name>Boris Springborn</name></author>
-  </entry>
-</feed>"""
-
-_ATOM_EMPTY = """<?xml version="1.0" encoding="UTF-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>ArXiv Query</title></feed>"""
-
-
-def test_fetch_metadata_parses_entry(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A valid Atom feed yields a populated Paper."""
-
-    def mock_get(url: str, **kwargs: object) -> httpx.Response:
-        return httpx.Response(200, text=_ATOM_FEED, request=httpx.Request("GET", url))
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-
-    paper = arxiv.fetch_metadata("1911.00966")
-    assert paper is not None
-    assert paper.id == "1911.00966"
-    assert paper.title.startswith("A discrete version of Liouville")
-    assert paper.authors == ["Ulrich Pinkall", "Boris Springborn"]
-    assert paper.year == 2019
-    assert paper.abstract and "conformal maps" in paper.abstract
-    assert paper.openalex_id is None
-
-
-def test_fetch_metadata_no_entry_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A feed with no <entry> (id not found) returns None."""
-
-    def mock_get(url: str, **kwargs: object) -> httpx.Response:
-        return httpx.Response(200, text=_ATOM_EMPTY, request=httpx.Request("GET", url))
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-    assert arxiv.fetch_metadata("0000.00000") is None
-
-
-def test_fetch_metadata_strips_arxiv_prefix(monkeypatch: pytest.MonkeyPatch) -> None:
-    """An ``arXiv:`` prefix is stripped so Paper.id is the bare id."""
-
-    def mock_get(url: str, **kwargs: object) -> httpx.Response:
-        return httpx.Response(200, text=_ATOM_FEED, request=httpx.Request("GET", url))
-
-    monkeypatch.setattr(httpx, "get", mock_get)
-    paper = arxiv.fetch_metadata("arXiv:1911.00966")
-    assert paper is not None
-    assert paper.id == "1911.00966"

tests/sources/test_crossref.py

@@ -1,104 +0,0 @@
-"""Tests for codex.sources.crossref."""
-
-from __future__ import annotations
-
-import httpx
-import pytest
-
-from codex.models import Citation
-from codex.sources import crossref
-
-
-def test_fetch_abstract_strips_jats(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A JATS-tagged abstract is returned as plain text with tags removed."""
-    body = {
-        "message": {
-            "abstract": "<jats:p>We prove a <jats:italic>theorem</jats:italic> about "
-            "polyhedra.</jats:p>"
-        }
-    }
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json=body)
-
-    monkeypatch.setattr(crossref, "_get", mock_get)
-
-    result = crossref.fetch_abstract("10.1007/s00454-019-00132-8")
-    assert result == "We prove a theorem about polyhedra."
-
-
-def test_fetch_abstract_absent_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A work with no abstract field (e.g. a book chapter) returns None."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json={"message": {"title": ["Some Chapter"]}})
-
-    monkeypatch.setattr(crossref, "_get", mock_get)
-    assert crossref.fetch_abstract("10.1007/978-3-642-17413-1_7") is None
-
-
-def test_fetch_abstract_404_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A 404 returns None rather than raising."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        raise httpx.HTTPStatusError(
-            "404",
-            request=httpx.Request("GET", url),
-            response=httpx.Response(404, request=httpx.Request("GET", url)),
-        )
-
-    monkeypatch.setattr(crossref, "_get", mock_get)
-    assert crossref.fetch_abstract("10.0000/nonexistent") is None
-
-
-# ---------------------------------------------------------------------------
-# fetch_references (roadmap R-B)
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_references_returns_doi_edges(monkeypatch: pytest.MonkeyPatch) -> None:
-    """Only references carrying a DOI become Citations; citing_id is the queried DOI."""
-    body = {
-        "message": {
-            "reference": [
-                {"key": "ref1", "DOI": "10.1090/s0002-9947-04-03545-7"},
-                {"key": "ref2", "unstructured": "A book with no DOI, 1992."},  # skipped
-                {"key": "ref3", "DOI": "10.1007/bf02392449", "article-title": "X"},
-            ]
-        }
-    }
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json=body)
-
-    monkeypatch.setattr(crossref, "_get", mock_get)
-    refs = crossref.fetch_references("10.1007/s00454-019-00132-8")
-
-    assert refs == [
-        Citation(citing_id="10.1007/s00454-019-00132-8", cited_id="10.1090/s0002-9947-04-03545-7"),
-        Citation(citing_id="10.1007/s00454-019-00132-8", cited_id="10.1007/bf02392449"),
-    ]
-
-
-def test_fetch_references_no_reference_key_returns_empty(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A work with no deposited reference list returns []."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json={"message": {"title": ["No refs deposited"]}})
-
-    monkeypatch.setattr(crossref, "_get", mock_get)
-    assert crossref.fetch_references("10.1007/978-3-642-17413-1_7") == []
-
-
-def test_fetch_references_404_returns_empty(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A 404 returns [] rather than raising."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        raise httpx.HTTPStatusError(
-            "404",
-            request=httpx.Request("GET", url),
-            response=httpx.Response(404, request=httpx.Request("GET", url)),
-        )
-
-    monkeypatch.setattr(crossref, "_get", mock_get)
-    assert crossref.fetch_references("10.0000/nonexistent") == []

tests/sources/test_openalex.py

@@ -1,289 +0,0 @@
-"""Tests for codex.sources.openalex."""
-
-from __future__ import annotations
-
-import httpx
-import pytest
-
-from codex.models import Citation, Paper
-from codex.sources import openalex
-from codex.sources.openalex import _canonical_id, _normalize_doi, _resolve_id
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-_SAMPLE_WORK = {
-    "id": "https://openalex.org/W2741809807",
-    # Real OpenAlex returns the DOI as a full URL, not a bare DOI (audit T-3 / DQ-5
-    # — the old bare fixture hid the URL-form behaviour both fixes address).
-    "doi": "https://doi.org/10.1145/3592430",
-    "title": "Conformal Prediction: A Review",
-    "publication_year": 2023,
-    "authorships": [
-        {"author": {"display_name": "Alice Smith"}},
-        {"author": {"display_name": "Bob Jones"}},
-    ],
-    "abstract_inverted_index": {
-        "Conformal": [0],
-        "prediction": [1],
-        "is": [2],
-        "great": [3],
-    },
-    "referenced_works": [
-        "https://openalex.org/W1",
-        "https://openalex.org/W2",
-    ],
-}
-
-
-# ---------------------------------------------------------------------------
-# _resolve_id
-# ---------------------------------------------------------------------------
-
-
-def test_resolve_doi() -> None:
-    assert _resolve_id("10.1145/3592430") == "doi:10.1145/3592430"
-
-
-def test_resolve_arxiv() -> None:
-    # arXiv IDs resolve to their arXiv DOI (the arxiv: path 404s on OpenAlex).
-    assert _resolve_id("2301.07041") == "doi:10.48550/arXiv.2301.07041"
-
-
-def test_resolve_arxiv_legacy() -> None:
-    # Legacy category/number IDs (pre-2007) resolve the same way.
-    assert _resolve_id("math/0603097") == "doi:10.48550/arXiv.math/0603097"
-
-
-def test_resolve_arxiv_prefix_stripped() -> None:
-    # An explicit arxiv: prefix is stripped before building the arXiv DOI.
-    assert _resolve_id("arxiv:2301.07041") == "doi:10.48550/arXiv.2301.07041"
-
-
-def test_resolve_w_id_unchanged() -> None:
-    assert _resolve_id("W2741809807") == "W2741809807"
-
-
-def test_resolve_prefixed_unchanged() -> None:
-    assert _resolve_id("doi:10.1145/x") == "doi:10.1145/x"
-
-
-# ---------------------------------------------------------------------------
-# _normalize_doi — canonical bare, lower-cased form (DQ-5)
-# ---------------------------------------------------------------------------
-
-
-def test_normalize_doi_strips_https_url() -> None:
-    # The exact form OpenAlex returns in the `doi` field.
-    assert _normalize_doi("https://doi.org/10.1007/s00454-019-00132-8") == (
-        "10.1007/s00454-019-00132-8"
-    )
-
-
-def test_normalize_doi_strips_http_and_doi_prefix() -> None:
-    assert _normalize_doi("http://doi.org/10.1145/3592430") == "10.1145/3592430"
-    assert _normalize_doi("doi:10.1145/3592430") == "10.1145/3592430"
-
-
-def test_normalize_doi_lowercases() -> None:
-    # DOIs are case-insensitive; the canonical id (and _norm_cited_id) is lower-case.
-    assert _normalize_doi("https://doi.org/10.1007/S00454-019-00132-8") == (
-        "10.1007/s00454-019-00132-8"
-    )
-
-
-def test_normalize_doi_already_bare_is_idempotent() -> None:
-    assert _normalize_doi("10.1007/s00454-019-00132-8") == "10.1007/s00454-019-00132-8"
-
-
-# ---------------------------------------------------------------------------
-# _canonical_id — arXiv DOI → bare arXiv id (arXiv half of DQ-5)
-# ---------------------------------------------------------------------------
-
-
-def test_canonical_id_strips_arxiv_doi_modern() -> None:
-    # arXiv works carry the arXiv DOI; the canonical id is the bare arXiv id.
-    assert _canonical_id("https://doi.org/10.48550/arXiv.2305.10988") == "2305.10988"
-
-
-def test_canonical_id_strips_arxiv_doi_legacy() -> None:
-    assert _canonical_id("https://doi.org/10.48550/arXiv.math/0603097") == "math/0603097"
-
-
-def test_canonical_id_leaves_regular_doi_bare() -> None:
-    # A normal journal DOI is only normalized (bare + lower-case), not stripped.
-    assert _canonical_id("https://doi.org/10.1007/S00454-019-00132-8") == (
-        "10.1007/s00454-019-00132-8"
-    )
-
-
-def test_fetch_paper_arxiv_doi_yields_bare_id(monkeypatch: pytest.MonkeyPatch) -> None:
-    """An arXiv work maps to the BARE arXiv id (not the arXiv DOI), so re-ingesting
-    by arXiv id hits ON CONFLICT (id) instead of tripping papers_openalex_id_key."""
-    work = {**_SAMPLE_WORK, "doi": "https://doi.org/10.48550/arXiv.2305.10988"}
-
-    def mock_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        return httpx.Response(200, json=work)
-
-    monkeypatch.setattr(openalex, "_get", mock_get)
-    paper = openalex.fetch_paper("2305.10988")
-
-    assert paper is not None
-    assert paper.id == "2305.10988"
-
-
-def test_map_paper_falls_back_to_openalex_id_without_doi(
-    monkeypatch: pytest.MonkeyPatch,
-) -> None:
-    """A work with no DOI keeps the OpenAlex W-id (URL form) as id, unchanged —
-    normalization only rewrites the DOI branch."""
-    work = {k: v for k, v in _SAMPLE_WORK.items() if k != "doi"}
-
-    def mock_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        return httpx.Response(200, json=work)
-
-    monkeypatch.setattr(openalex, "_get", mock_get)
-    paper = openalex.fetch_paper("W2741809807")
-
-    assert paper is not None
-    assert paper.id == "https://openalex.org/W2741809807"
-
-
-# ---------------------------------------------------------------------------
-# fetch_paper — success
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_paper_success(monkeypatch: pytest.MonkeyPatch) -> None:
-    """Successful fetch maps all fields to Paper correctly."""
-
-    def mock_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        assert "doi:10.1145/3592430" in url
-        return httpx.Response(200, json=_SAMPLE_WORK)
-
-    monkeypatch.setattr(openalex, "_get", mock_get)
-
-    paper = openalex.fetch_paper("10.1145/3592430")
-
-    assert paper is not None
-    assert isinstance(paper, Paper)
-    # DQ-5: openalex._canonical_id normalizes OpenAlex's URL-form doi to the BARE
-    # lower-cased DOI at the source layer; ingest additionally pins papers.id to the
-    # caller's id (audit C-7). Both converge on the bare canonical id — which is what
-    # makes the ingest ON CONFLICT (id) upsert idempotent. (T-3: the fixture is
-    # URL-form so this mapping can't silently regress.)
-    assert paper.id == "10.1145/3592430"
-    assert paper.title == "Conformal Prediction: A Review"
-    assert paper.year == 2023
-    assert paper.authors == ["Alice Smith", "Bob Jones"]
-    assert paper.abstract == "Conformal prediction is great"
-    assert paper.openalex_id == "https://openalex.org/W2741809807"
-
-
-def test_fetch_paper_arxiv_id_prefixed(monkeypatch: pytest.MonkeyPatch) -> None:
-    """Bare arXiv IDs are resolved to their arXiv DOI in the URL."""
-    called_url: list[str] = []
-
-    def mock_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        called_url.append(url)
-        return httpx.Response(200, json=_SAMPLE_WORK)
-
-    monkeypatch.setattr(openalex, "_get", mock_get)
-    openalex.fetch_paper("2301.07041")
-
-    assert called_url and "doi:10.48550/arXiv.2301.07041" in called_url[0]
-
-
-# ---------------------------------------------------------------------------
-# fetch_paper — 404 → None
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_paper_404_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A 404 response should return None without raising."""
-
-    def mock_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        raise httpx.HTTPStatusError(
-            "Not Found",
-            request=httpx.Request("GET", url),
-            response=httpx.Response(404, request=httpx.Request("GET", url)),
-        )
-
-    monkeypatch.setattr(openalex, "_get", mock_get)
-
-    result = openalex.fetch_paper("nonexistent-id")
-    assert result is None
-
-
-# ---------------------------------------------------------------------------
-# fetch_paper — 429 retry then success
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_paper_retry_on_429(monkeypatch: pytest.MonkeyPatch) -> None:
-    """On 429×3 then 200, returns 1 Paper after 3 failed attempts."""
-    attempt = [0]
-
-    def controlled_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        attempt[0] += 1
-        if attempt[0] <= 3:
-            resp = httpx.Response(429, request=httpx.Request("GET", url))
-            raise httpx.HTTPStatusError(
-                "Too Many Requests",
-                request=httpx.Request("GET", url),
-                response=resp,
-            )
-        return httpx.Response(200, json=_SAMPLE_WORK)
-
-    monkeypatch.setattr(openalex, "_get", controlled_get)
-
-    result: Paper | None = None
-    errors = 0
-    for _ in range(5):
-        try:
-            result = openalex.fetch_paper("10.1145/3592430")
-            break
-        except httpx.HTTPStatusError:
-            errors += 1
-
-    assert result is not None
-    assert isinstance(result, Paper)
-    assert errors == 3
-
-
-# ---------------------------------------------------------------------------
-# fetch_citations — uses referenced_works field
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_citations(monkeypatch: pytest.MonkeyPatch) -> None:
-    """fetch_citations reads referenced_works from the work object."""
-
-    def mock_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        return httpx.Response(200, json=_SAMPLE_WORK)
-
-    monkeypatch.setattr(openalex, "_get", mock_get)
-
-    oa_id = "https://openalex.org/W2741809807"
-    citations = openalex.fetch_citations(oa_id)
-
-    assert len(citations) == 2
-    assert all(isinstance(c, Citation) for c in citations)
-    assert citations[0].citing_id == oa_id
-    assert citations[0].cited_id == "https://openalex.org/W1"
-    assert citations[1].cited_id == "https://openalex.org/W2"
-
-
-def test_fetch_citations_404_returns_empty(monkeypatch: pytest.MonkeyPatch) -> None:
-    """404 on the work fetch returns an empty list."""
-
-    def mock_get(url: str, params: dict[str, str] | None = None) -> httpx.Response:
-        raise httpx.HTTPStatusError(
-            "Not Found",
-            request=httpx.Request("GET", url),
-            response=httpx.Response(404, request=httpx.Request("GET", url)),
-        )
-
-    monkeypatch.setattr(openalex, "_get", mock_get)
-    assert openalex.fetch_citations("W999") == []

tests/sources/test_semanticscholar.py

@@ -1,169 +0,0 @@
-"""Tests for codex.sources.semanticscholar."""
-
-from __future__ import annotations
-
-import httpx
-import pytest
-
-from codex.models import Citation
-from codex.sources import semanticscholar
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-_SAMPLE_REFS_RESPONSE = {
-    "data": [
-        {
-            "citedPaper": {
-                "paperId": "abc123",
-                "externalIds": {"DOI": "10.1000/xyz1"},
-            },
-            "contexts": ["This approach was first described in [1]."],
-        },
-        {
-            "citedPaper": {
-                "paperId": "def456",
-                "externalIds": {"ArXiv": "2301.07041"},
-            },
-            "contexts": [],
-        },
-    ]
-}
-
-_SAMPLE_RECS_RESPONSE = {
-    "recommendedPapers": [
-        {"paperId": "rec_id_1"},
-        {"paperId": "rec_id_2"},
-        {"paperId": "rec_id_3"},
-    ]
-}
-
-
-# ---------------------------------------------------------------------------
-# fetch_references
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_references_returns_citations(monkeypatch: pytest.MonkeyPatch) -> None:
-    """fetch_references maps API response to Citation objects correctly."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json=_SAMPLE_REFS_RESPONSE)
-
-    monkeypatch.setattr(semanticscholar, "_get", mock_get)
-
-    citations = semanticscholar.fetch_references("abc123")
-
-    assert len(citations) == 2
-    assert all(isinstance(c, Citation) for c in citations)
-
-    # First citation uses DOI and has context
-    assert citations[0].citing_id == "abc123"
-    assert citations[0].cited_id == "10.1000/xyz1"
-    assert citations[0].context == "This approach was first described in [1]."
-
-    # Second citation falls back to ArXiv ID, no context
-    assert citations[1].citing_id == "abc123"
-    assert citations[1].cited_id == "2301.07041"
-    assert citations[1].context is None
-
-
-def test_fetch_references_404_returns_empty(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A 404 response should return an empty list."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        raise httpx.HTTPStatusError(
-            "Not Found",
-            request=httpx.Request("GET", url),
-            response=httpx.Response(404, request=httpx.Request("GET", url)),
-        )
-
-    monkeypatch.setattr(semanticscholar, "_get", mock_get)
-
-    result = semanticscholar.fetch_references("nonexistent")
-    assert result == []
-
-
-# ---------------------------------------------------------------------------
-# fetch_recommendations
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_recommendations_returns_list_of_str(
-    monkeypatch: pytest.MonkeyPatch,
-) -> None:
-    """fetch_recommendations returns a flat list of paperId strings."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json=_SAMPLE_RECS_RESPONSE)
-
-    monkeypatch.setattr(semanticscholar, "_get", mock_get)
-
-    result = semanticscholar.fetch_recommendations("abc123", limit=3)
-
-    assert isinstance(result, list)
-    assert len(result) == 3
-    assert all(isinstance(r, str) for r in result)
-    assert result == ["rec_id_1", "rec_id_2", "rec_id_3"]
-
-
-def test_fetch_recommendations_404_returns_empty(
-    monkeypatch: pytest.MonkeyPatch,
-) -> None:
-    """A 404 response should return an empty list."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        raise httpx.HTTPStatusError(
-            "Not Found",
-            request=httpx.Request("GET", url),
-            response=httpx.Response(404, request=httpx.Request("GET", url)),
-        )
-
-    monkeypatch.setattr(semanticscholar, "_get", mock_get)
-
-    result = semanticscholar.fetch_recommendations("nonexistent")
-    assert result == []
-
-
-# ---------------------------------------------------------------------------
-# fetch_references — {"data": null} guard (DQ-1 regression)
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_references_null_data_returns_empty(monkeypatch: pytest.MonkeyPatch) -> None:
-    """S2 returns HTTP 200 with ``{"data": null}`` for a known paper that has no
-    parsed references; this must yield [] rather than raising TypeError."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json={"data": None, "citingPaperInfo": {}})
-
-    monkeypatch.setattr(semanticscholar, "_get", mock_get)
-    assert semanticscholar.fetch_references("DOI:10.14279/depositonce-5415") == []
-
-
-# ---------------------------------------------------------------------------
-# fetch_abstract (DQ-2)
-# ---------------------------------------------------------------------------
-
-
-def test_fetch_abstract_returns_text(monkeypatch: pytest.MonkeyPatch) -> None:
-    """fetch_abstract returns the abstract string when present."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json={"abstract": "We study ideal hyperbolic polyhedra."})
-
-    monkeypatch.setattr(semanticscholar, "_get", mock_get)
-    assert semanticscholar.fetch_abstract("DOI:10.1007/s00454-019-00132-8") == (
-        "We study ideal hyperbolic polyhedra."
-    )
-
-
-def test_fetch_abstract_null_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    """A null/absent abstract yields None."""
-
-    def mock_get(url: str, params: dict[str, object] | None = None) -> httpx.Response:
-        return httpx.Response(200, json={"abstract": None})
-
-    monkeypatch.setattr(semanticscholar, "_get", mock_get)
-    assert semanticscholar.fetch_abstract("arXiv:1234.5678") is None

tests/synthesis/conftest.py

@@ -1,124 +0,0 @@
-"""Shared synthesis fixtures — keep DB/LLM/embed fully mocked."""
-
-from __future__ import annotations
-
-from pathlib import Path
-from typing import Any
-from unittest.mock import MagicMock
-
-import pytest
-
-# Shared corpus fixture used across the test suite.
-FIXTURE_CHUNKS: list[dict[str, Any]] = [
-    {
-        "id": 1,
-        "paper_id": "springborn-2008",
-        "ord": 16,
-        "content": (
-            "For an ideal tetrahedron with dihedral angles the hyperbolic volume is "
-            "V = L(gamma1) + L(gamma2) + L(gamma3). "
-            "The volume function V0 is strictly concave on the angle domain."
-        ),
-        "bibkey": "Springborn2008",
-    },
-    {
-        "id": 2,
-        "paper_id": "bps-2015",
-        "ord": 3,
-        "content": (
-            "A discrete conformal map is defined by logarithmic scale factors u_i "
-            "at each vertex such that the edge lengths satisfy a compatibility condition."
-        ),
-        "bibkey": "BobenkoPinkallSpringborn2015",
-    },
-    {
-        "id": 3,
-        "paper_id": "luo-2004",
-        "ord": 0,
-        "content": (
-            "The combinatorial Yamabe flow drives the edge lengths towards a target "
-            "curvature on a triangulated surface."
-        ),
-        "bibkey": "Luo2004",
-    },
-]
-
-
-class StubLLM:
-    """Deterministic mock LLM that returns a preset response."""
-
-    def __init__(self, response: str | list[str]) -> None:
-        if isinstance(response, str):
-            self._responses = [response]
-        else:
-            self._responses = list(response)
-        self._calls: list[tuple[str, str]] = []
-
-    def generate(self, prompt: str, model: str) -> str:
-        self._calls.append((prompt, model))
-        if not self._responses:
-            return ""
-        if len(self._responses) == 1:
-            return self._responses[0]
-        return self._responses.pop(0)
-
-    @property
-    def calls(self) -> list[tuple[str, str]]:
-        return list(self._calls)
-
-
-@pytest.fixture
-def stub_llm() -> StubLLM:
-    """Default stub returning empty string (override per test as needed)."""
-    return StubLLM("")
-
-
-@pytest.fixture
-def fixture_chunks() -> list[dict[str, Any]]:
-    return [dict(c) for c in FIXTURE_CHUNKS]
-
-
-@pytest.fixture
-def mock_retrieve(monkeypatch: pytest.MonkeyPatch) -> None:
-    """Patch _retrieve_chunks to return deterministic fixture chunks (no DB)."""
-    monkeypatch.setattr(
-        "codex.synthesis._retrieve_chunks",
-        lambda queries, top_k: [dict(c) for c in FIXTURE_CHUNKS],
-    )
-
-
-@pytest.fixture
-def mock_paper_titles(monkeypatch: pytest.MonkeyPatch) -> None:
-    """Patch _list_paper_titles to return a deterministic three-paper corpus."""
-    monkeypatch.setattr(
-        "codex.synthesis._list_paper_titles",
-        lambda db_conn: [
-            ("Springborn2008", "A variational principle for weighted Delaunay triangulations"),
-            (
-                "BobenkoPinkallSpringborn2015",
-                "Discrete conformal maps and ideal hyperbolic polyhedra",
-            ),
-            ("Luo2004", "Combinatorial Yamabe Flow on Surfaces"),
-        ],
-    )
-
-
-@pytest.fixture
-def mock_settings(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Path:
-    """Settings stub pointing leads_dir at tmp_path/leads."""
-    from codex.config import Settings
-
-    leads_path = tmp_path / "leads"
-
-    mock = MagicMock(spec=Settings)
-    mock.leads_dir = str(leads_path)
-    mock.synthesis_llm_model = "test-model"
-    mock.synthesis_llm_url = None
-    mock.ollama_base_url = "http://localhost:11434"
-    mock.synthesis_top_k = 6
-    mock.synthesis_min_grounded_ratio = 0.5
-    mock.synthesis_gap_min_coverage = 2
-    mock.wiki_dir = str(tmp_path / "wiki")
-
-    monkeypatch.setattr("codex.synthesis.get_settings", lambda: mock)
-    return leads_path

tests/synthesis/test_cli.py

@@ -1,177 +0,0 @@
-"""Tests for the ``codex synthesis`` CLI command group."""
-
-from __future__ import annotations
-
-from pathlib import Path
-from unittest.mock import MagicMock
-
-import pytest
-from typer.testing import CliRunner
-
-from codex.cli import app
-from codex.synthesis import Lead, Provenance
-
-runner = CliRunner()
-
-
-def _grounded_lead(lead_id: str = "L-0001") -> Lead:
-    return Lead(
-        id=lead_id,
-        kind="connection",
-        title="Connection — X ⇄ Y",
-        body="Some grounded body. [X2020 #chunk 1]",
-        provenance=[Provenance(bibkey="X2020", locator="chunk 1")],
-        confidence=0.8,
-    )
-
-
-def _conjecture(lead_id: str = "L-0002") -> Lead:
-    return Lead(
-        id=lead_id,
-        kind="conjecture",
-        title="Bold guess",
-        body="Hypothetical.",
-        provenance=[Provenance(bibkey="X2020", locator="chunk 1")],
-        confidence=0.0,
-        suggested_validation="Run a falsification spike.",
-    )
-
-
-@pytest.fixture
-def isolated_leads_dir(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Path:
-    """Point Settings.leads_dir at tmp_path/leads for CLI tests."""
-    from codex.config import Settings
-
-    leads_path = tmp_path / "leads"
-
-    settings = MagicMock(spec=Settings)
-    settings.leads_dir = str(leads_path)
-    settings.synthesis_llm_model = "test-model"
-    settings.synthesis_llm_url = None
-    settings.ollama_base_url = "http://localhost:11434"
-    settings.synthesis_top_k = 6
-    settings.synthesis_min_grounded_ratio = 0.5
-    settings.synthesis_gap_min_coverage = 2
-
-    monkeypatch.setattr("codex.cli.get_settings", lambda: settings, raising=False)
-    monkeypatch.setattr("codex.synthesis.get_settings", lambda: settings)
-    monkeypatch.setattr("codex.config.get_settings", lambda: settings, raising=False)
-    return leads_path
-
-
-def test_synthesis_leads_invokes_three_finders(
-    monkeypatch: pytest.MonkeyPatch,
-    isolated_leads_dir: Path,
-) -> None:
-    """`codex synthesis leads --lib-path` calls connection + gap + improvement."""
-    calls: list[str] = []
-
-    def fake_connections(*, top_k: int, llm: object) -> list[Lead]:
-        calls.append("connection")
-        return [_grounded_lead("L-0001")]
-
-    def fake_gaps(
-        *,
-        lib_path: str | None = None,
-        llm: object,
-        topics: list[str] | None = None,
-    ) -> list[Lead]:
-        calls.append("gap")
-        return []
-
-    def fake_improvements(lib_path: str, *, top_k: int, llm: object) -> list[Lead]:
-        calls.append("improvement")
-        return []
-
-    monkeypatch.setattr("codex.synthesis.find_connections", fake_connections)
-    monkeypatch.setattr("codex.synthesis.find_gaps", fake_gaps)
-    monkeypatch.setattr("codex.synthesis.find_improvements", fake_improvements)
-    monkeypatch.setattr("codex.synthesis.default_llm_client", lambda: object())
-
-    result = runner.invoke(app, ["synthesis", "leads", "--lib-path", "/tmp/fake"])
-
-    assert result.exit_code == 0, result.stdout
-    assert calls == ["connection", "gap", "improvement"]
-    assert (isolated_leads_dir / "grounded" / "L-0001.md").exists()
-
-
-def test_synthesis_leads_without_lib_path_skips_improvement(
-    monkeypatch: pytest.MonkeyPatch,
-    isolated_leads_dir: Path,
-) -> None:
-    """Without --lib-path, the improvement finder must be skipped (graceful)."""
-    calls: list[str] = []
-
-    monkeypatch.setattr(
-        "codex.synthesis.find_connections",
-        lambda *, top_k, llm: (calls.append("connection"), [])[1],
-    )
-    monkeypatch.setattr(
-        "codex.synthesis.find_gaps",
-        lambda *, lib_path=None, llm, topics=None: (calls.append("gap"), [])[1],
-    )
-    monkeypatch.setattr(
-        "codex.synthesis.find_improvements",
-        lambda *args, **kwargs: (calls.append("improvement"), [])[1],
-    )
-    monkeypatch.setattr("codex.synthesis.default_llm_client", lambda: object())
-
-    result = runner.invoke(app, ["synthesis", "leads"])
-
-    assert result.exit_code == 0, result.stdout
-    assert "improvement" not in calls
-    assert "connection" in calls
-    assert "gap" in calls
-
-
-def test_synthesis_conjectures_writes_to_conjectures_dir(
-    monkeypatch: pytest.MonkeyPatch,
-    isolated_leads_dir: Path,
-) -> None:
-    """`codex synthesis conjectures` lands files under leads/conjectures/."""
-    monkeypatch.setattr(
-        "codex.synthesis.propose_conjectures",
-        lambda *, top_k, llm: [_conjecture("L-0042")],
-    )
-    monkeypatch.setattr("codex.synthesis.default_llm_client", lambda: object())
-
-    result = runner.invoke(app, ["synthesis", "conjectures"])
-
-    assert result.exit_code == 0, result.stdout
-    assert (isolated_leads_dir / "conjectures" / "L-0042.md").exists()
-    assert not (isolated_leads_dir / "grounded" / "L-0042.md").exists()
-
-
-def test_synthesis_report_separates_sections(
-    monkeypatch: pytest.MonkeyPatch,
-    isolated_leads_dir: Path,
-) -> None:
-    """`codex synthesis report` shows grounded and conjectures in distinct headers."""
-    from codex.synthesis import write_leads
-
-    write_leads(
-        [_grounded_lead("L-0001"), _conjecture("L-0002")],
-        str(isolated_leads_dir),
-    )
-
-    result = runner.invoke(app, ["synthesis", "report"])
-
-    assert result.exit_code == 0, result.stdout
-    assert "GROUNDED LEADS" in result.stdout
-    assert "CONJECTURES" in result.stdout
-    # Visible separation: GROUNDED header strictly precedes CONJECTURES header
-    assert result.stdout.index("GROUNDED LEADS") < result.stdout.index("CONJECTURES")
-    # UNVERIFIED tag in the conjectures header
-    assert "UNVERIFIED" in result.stdout
-
-
-def test_synthesis_report_empty(
-    isolated_leads_dir: Path,
-) -> None:
-    """`codex synthesis report` with no leads still prints both sections."""
-    result = runner.invoke(app, ["synthesis", "report"])
-
-    assert result.exit_code == 0
-    assert "GROUNDED LEADS" in result.stdout
-    assert "CONJECTURES" in result.stdout
-    assert "(none)" in result.stdout

tests/synthesis/test_conjecture.py

@@ -1,113 +0,0 @@
-"""Tests for stage-5 conjecture generation.
-
-HARD invariant: every conjecture MUST carry status='unverified', a
-non-empty suggested_validation, and non-empty provenance.
-"""
-
-from __future__ import annotations
-
-from pathlib import Path
-
-from codex.synthesis import _parse_conjecture_output, propose_conjectures
-from tests.synthesis.conftest import StubLLM
-
-_VALID_CONJECTURE = (
-    "Title: A discrete cosmetic-surgery analogue exists for Yamabe flows.\n"
-    "Body: The combinatorial Yamabe flow [Luo2004 #chunk 0] is a discrete "
-    "analogue of conformal change. One could hope the cosmetic surgery "
-    "conjecture has a discrete formulation in this setting.\n"
-    "Validation: Search for a counter-example among simplicial 3-manifolds "
-    "with at most 12 tetrahedra using a CSP solver."
-)
-
-_MISSING_VALIDATION = (
-    "Title: A naive idea.\n"
-    "Body: Some thoughts that follow [Luo2004 #chunk 0]. No falsification path "
-    "is described."
-)
-
-
-def test_parse_valid_conjecture_output() -> None:
-    """_parse_conjecture_output extracts title / body / validation."""
-    parsed = _parse_conjecture_output(_VALID_CONJECTURE)
-    assert parsed is not None
-    title, body, validation = parsed
-    assert title.startswith("A discrete cosmetic-surgery")
-    assert "Luo2004" in body
-    assert "counter-example" in validation
-
-
-def test_parse_conjecture_missing_validation_returns_none() -> None:
-    """Output without a Validation: line is rejected."""
-    assert _parse_conjecture_output(_MISSING_VALIDATION) is None
-
-
-def test_parse_conjecture_empty_input() -> None:
-    """Empty LLM output → None."""
-    assert _parse_conjecture_output("") is None
-
-
-def test_propose_conjectures_marks_status_unverified(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """Every emitted conjecture has status='unverified'."""
-    llm = StubLLM(_VALID_CONJECTURE)
-    leads = propose_conjectures(top_k=6, llm=llm)
-    assert len(leads) >= 1
-    for lead in leads:
-        assert lead.kind == "conjecture"
-        assert lead.status == "unverified"
-
-
-def test_propose_conjectures_requires_validation(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """Every emitted conjecture has a non-empty suggested_validation."""
-    llm = StubLLM(_VALID_CONJECTURE)
-    leads = propose_conjectures(top_k=6, llm=llm)
-    assert len(leads) >= 1
-    for lead in leads:
-        assert lead.suggested_validation.strip() != ""
-
-
-def test_propose_conjectures_requires_provenance(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """Every conjecture has at least one Provenance entry (seed chunks)."""
-    llm = StubLLM(_VALID_CONJECTURE)
-    leads = propose_conjectures(top_k=6, llm=llm)
-    assert len(leads) >= 1
-    for lead in leads:
-        assert len(lead.provenance) >= 1
-
-
-def test_propose_conjectures_drops_when_validation_missing(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """LLM output without a Validation: line is rejected (zero leads)."""
-    llm = StubLLM(_MISSING_VALIDATION)
-    leads = propose_conjectures(top_k=6, llm=llm)
-    assert leads == []
-
-
-def test_propose_conjectures_handles_llm_failure(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """LLM raising → returns [] (mirrors F-12 graceful degradation)."""
-
-    class FailingLLM:
-        def generate(self, prompt: str, model: str) -> str:
-            raise ConnectionError("ollama unreachable")
-
-    leads = propose_conjectures(top_k=6, llm=FailingLLM())
-    assert leads == []

tests/synthesis/test_gaps.py

@@ -1,161 +0,0 @@
-"""Tests for find_gaps — coverage map + code-vs-corpus signals."""
-
-from __future__ import annotations
-
-from pathlib import Path
-from typing import Any
-
-import pytest
-
-from codex.synthesis import find_gaps
-from tests.synthesis.conftest import StubLLM
-
-
-def test_find_gaps_code_cites_missing_bibkey(
-    monkeypatch: pytest.MonkeyPatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-    fixture_chunks: list[dict[str, object]],
-    tmp_path: Path,
-) -> None:
-    """A bibkey referenced in code but absent from corpus → direct gap lead."""
-    # Build a small C++ source tree with an @cite pointing at a missing bibkey
-    src_dir = tmp_path / "src"
-    src_dir.mkdir()
-    (src_dir / "main.cpp").write_text(
-        "// @cite UnknownPaper2099\nvoid foo() {}\n",
-        encoding="utf-8",
-    )
-
-    # All probes return the fixture chunks (so coverage map yields nothing)
-    monkeypatch.setattr(
-        "codex.synthesis._retrieve_chunks",
-        lambda queries, top_k: [dict(c) for c in fixture_chunks],
-    )
-
-    leads = find_gaps(lib_path=str(src_dir), llm=StubLLM(""))
-    bibkeys_in_gaps = {p.bibkey for lead in leads for p in lead.provenance}
-    # The missing bibkey shows up as a gap
-    assert any("UnknownPaper2099" in lead.title for lead in leads)
-    assert "UnknownPaper2099" in bibkeys_in_gaps
-
-
-def test_find_gaps_code_known_bibkey_not_reported(
-    monkeypatch: pytest.MonkeyPatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-    fixture_chunks: list[dict[str, object]],
-    tmp_path: Path,
-) -> None:
-    """A bibkey already in the corpus is NOT flagged by the code-vs-corpus path."""
-    src_dir = tmp_path / "src"
-    src_dir.mkdir()
-    (src_dir / "main.cpp").write_text(
-        "// @cite Springborn2008\nvoid foo() {}\n",
-        encoding="utf-8",
-    )
-    monkeypatch.setattr(
-        "codex.synthesis._retrieve_chunks",
-        lambda queries, top_k: [dict(c) for c in fixture_chunks],
-    )
-    leads = find_gaps(lib_path=str(src_dir), llm=StubLLM(""))
-    # Springborn2008 IS in the corpus → not a gap
-    assert not any("Springborn2008" in lead.title for lead in leads)
-
-
-def test_find_gaps_missing_lib_path_ok(
-    monkeypatch: pytest.MonkeyPatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-    fixture_chunks: list[dict[str, object]],
-) -> None:
-    """Without lib_path the code-vs-corpus path is skipped (no crash)."""
-    monkeypatch.setattr(
-        "codex.synthesis._retrieve_chunks",
-        lambda queries, top_k: [dict(c) for c in fixture_chunks],
-    )
-    leads = find_gaps(lib_path=None, llm=StubLLM(""))
-    # No crash; result list may be empty depending on coverage map.
-    assert isinstance(leads, list)
-
-
-def test_find_gaps_coverage_map_low_coverage_triggers_llm(
-    monkeypatch: pytest.MonkeyPatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """A topic covered by only 1 bibkey (< 2) triggers the LLM gap prompt.
-
-    The grounded body must reference Springborn2008 [...] in a content-5-gram
-    that exists in the chunk, otherwise the lead is dropped.
-    """
-    sparse_chunks: list[dict[str, Any]] = [
-        {
-            "id": 10,
-            "paper_id": "springborn-2008",
-            "ord": 16,
-            "content": (
-                "For an ideal tetrahedron with dihedral angles the hyperbolic volume is "
-                "V = L(gamma1) + L(gamma2) + L(gamma3). "
-                "The volume function V0 is strictly concave on the angle domain."
-            ),
-            "bibkey": "Springborn2008",
-        },
-    ]
-    monkeypatch.setattr(
-        "codex.synthesis._retrieve_chunks",
-        lambda queries, top_k: [dict(c) for c in sparse_chunks],
-    )
-    grounded_body = (
-        "Only Springborn2008 covers the volume formula at depth, while comparison "
-        "papers are missing.\n"
-        "For an ideal tetrahedron with dihedral angles the hyperbolic volume is "
-        "V = L(gamma1) + L(gamma2) + L(gamma3). [Springborn2008 #chunk 16]\n"
-    )
-    # Coverage gaps are opt-in (audit R-8): pass the topic explicitly.
-    leads = find_gaps(llm=StubLLM(grounded_body), topics=["hyperbolic volume formula"])
-    assert any(lead.kind == "gap" for lead in leads)
-
-
-def test_find_gaps_no_coverage_gaps_without_explicit_topics(
-    monkeypatch: pytest.MonkeyPatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """Without explicit topics, paper titles do NOT auto-generate coverage gaps (R-8)."""
-    sparse_chunks: list[dict[str, Any]] = [
-        {
-            "id": 10,
-            "paper_id": "springborn-2008",
-            "ord": 16,
-            "content": "the hyperbolic volume is V = L(gamma1) + L(gamma2) + L(gamma3)",
-            "bibkey": "Springborn2008",
-        },
-    ]
-    monkeypatch.setattr(
-        "codex.synthesis._retrieve_chunks",
-        lambda queries, top_k: [dict(c) for c in sparse_chunks],
-    )
-    # No lib_path, no topics → the coverage map must not run despite sparse coverage.
-    leads = find_gaps(llm=StubLLM("a gap. [Springborn2008 #chunk 16]"))
-    assert leads == []
-
-
-def test_find_gaps_explicit_topics_override_default(
-    monkeypatch: pytest.MonkeyPatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """Passing topics= explicitly overrides the default (paper titles)."""
-    captured_queries: list[list[str]] = []
-
-    def fake_retrieve(queries: list[str], top_k: int) -> list[dict[str, Any]]:
-        captured_queries.append(queries)
-        return []  # forces direct-gap-no-chunks path
-
-    monkeypatch.setattr("codex.synthesis._retrieve_chunks", fake_retrieve)
-    find_gaps(llm=StubLLM(""), topics=["custom topic A", "custom topic B"])
-    # Both custom topics should have been probed
-    flat = [q for qs in captured_queries for q in qs]
-    assert "custom topic A" in flat
-    assert "custom topic B" in flat

tests/synthesis/test_grounded.py

@@ -1,176 +0,0 @@
-"""Tests for grounded leads (connection / gap / improvement).
-
-The Grounding-Guard is re-used from :mod:`codex.wiki` (F-12). These tests
-verify that ungrounded claims are either dropped or annotated with ⚠,
-and that grounded statements survive intact.
-"""
-
-from __future__ import annotations
-
-from pathlib import Path
-
-from codex.synthesis import (
-    _finalise_grounded_lead,
-    _mark_ungrounded,
-    find_connections,
-    find_gaps,
-)
-from tests.synthesis.conftest import StubLLM
-
-_GROUNDED_CONNECTION = (
-    "Both papers build hyperbolic volumes from the Lobachevsky function.\n"
-    "For an ideal tetrahedron with dihedral angles the hyperbolic volume is "
-    "V = L(gamma1) + L(gamma2) + L(gamma3). [Springborn2008 #chunk 16]\n"
-    "A discrete conformal map is defined by logarithmic scale factors u_i at "
-    "each vertex such that the edge lengths satisfy a compatibility condition. "
-    "[BobenkoPinkallSpringborn2015 #chunk 3]\n"
-)
-
-_UNGROUNDED_CONNECTION = (
-    "Both papers prove the Riemann hypothesis via the Lobachevsky function. "
-    "[Springborn2008 #chunk 99]\n"
-)
-
-
-def test_finalise_grounded_lead_returns_none_without_claims(
-    fixture_chunks: list[dict[str, object]],
-) -> None:
-    """Body with no inline citations yields no grounded lead."""
-    lead = _finalise_grounded_lead(
-        seq=1,
-        kind="connection",
-        title="t",
-        body="A plain statement with no citation marker.",
-        chunks=fixture_chunks,
-        min_grounded_ratio=0.5,
-    )
-    assert lead is None
-
-
-def test_finalise_grounded_lead_drops_when_ratio_below_floor(
-    fixture_chunks: list[dict[str, object]],
-) -> None:
-    """All-ungrounded body → None (not silently promoted)."""
-    lead = _finalise_grounded_lead(
-        seq=1,
-        kind="connection",
-        title="t",
-        body=_UNGROUNDED_CONNECTION,
-        chunks=fixture_chunks,
-        min_grounded_ratio=0.5,
-    )
-    assert lead is None
-
-
-def test_finalise_grounded_lead_succeeds_when_grounded(
-    fixture_chunks: list[dict[str, object]],
-) -> None:
-    """Grounded body passes the floor and returns a populated Lead."""
-    lead = _finalise_grounded_lead(
-        seq=1,
-        kind="connection",
-        title="Both papers share a building block",
-        body=_GROUNDED_CONNECTION,
-        chunks=fixture_chunks,
-        min_grounded_ratio=0.5,
-    )
-    assert lead is not None
-    assert lead.kind == "connection"
-    assert lead.id == "L-C-0001"  # kind-prefixed id (audit C-11)
-    assert lead.confidence >= 0.5
-    assert lead.status == "unverified"
-    assert lead.suggested_validation == ""
-    # All claims should be grounded
-    assert all(c.grounded for c in lead.claims)
-
-
-def test_mark_ungrounded_prefixes_warning() -> None:
-    """_mark_ungrounded prefixes ⚠ to the matching claim text once."""
-    from codex.wiki import Claim
-
-    body = "Some fact. [X2020 #c 1]\nOther fact. [Y2021 #c 2]\n"
-    claims = [
-        Claim(text="Some fact", bibkey="X2020", locator="c 1", grounded=False),
-        Claim(text="Other fact", bibkey="Y2021", locator="c 2", grounded=True),
-    ]
-    out = _mark_ungrounded(body, claims)
-    assert "⚠ Some fact" in out
-    # Grounded claim is not annotated
-    assert "⚠ Other fact" not in out
-
-
-def test_find_connections_emits_grounded_lead(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """End-to-end: with a grounded LLM response, a connection lead is produced."""
-    llm = StubLLM(_GROUNDED_CONNECTION)
-    leads = find_connections(top_k=6, llm=llm)
-    assert len(leads) >= 1
-    lead = leads[0]
-    assert lead.kind == "connection"
-    assert lead.provenance, "grounded lead must carry provenance"
-    # No ⚠ should appear because all claims grounded
-    assert "⚠" not in lead.body
-
-
-def test_find_connections_drops_ungrounded_response(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """Ungrounded LLM responses are dropped, not emitted with ⚠."""
-    llm = StubLLM(_UNGROUNDED_CONNECTION)
-    leads = find_connections(top_k=6, llm=llm)
-    assert leads == []
-
-
-def test_find_connections_handles_llm_failure(
-    mock_retrieve: None,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """LLM raising → returns [] (mirrors F-12 graceful degradation)."""
-
-    class FailingLLM:
-        def generate(self, prompt: str, model: str) -> str:
-            raise ConnectionError("ollama unreachable")
-
-    leads = find_connections(top_k=6, llm=FailingLLM())
-    assert leads == []
-
-
-def test_find_gaps_topic_with_no_chunks(
-    monkeypatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-) -> None:
-    """When retrieval returns no chunks, a gap lead is emitted directly."""
-    monkeypatch.setattr("codex.synthesis._retrieve_chunks", lambda queries, top_k: [])
-    # Coverage gaps are opt-in now (audit R-8): pass an explicit topic.
-    leads = find_gaps(llm=StubLLM(""), topics=["nonexistent topic"])
-    assert len(leads) >= 1
-    assert all(lead.kind == "gap" for lead in leads)
-    # Direct gap leads carry a validation hint (re-ingest)
-    assert all(lead.suggested_validation for lead in leads)
-
-
-def test_find_gaps_well_covered_topic_skipped(
-    monkeypatch,
-    mock_paper_titles: None,
-    mock_settings: Path,
-    fixture_chunks: list[dict[str, object]],
-) -> None:
-    """Topics covered by enough bibkeys are NOT emitted as gaps.
-
-    Three distinct bibkeys in the fixture > synthesis_gap_min_coverage=2,
-    so every topic is well-covered and gets skipped.
-    """
-    monkeypatch.setattr(
-        "codex.synthesis._retrieve_chunks",
-        lambda queries, top_k: [dict(c) for c in fixture_chunks],
-    )
-    leads = find_gaps(llm=StubLLM(""))
-    # All probes well-covered → no gap leads
-    assert leads == []

tests/synthesis/test_model.py

@@ -1,112 +0,0 @@
-"""Tests for the Lead / Provenance datamodel (F-13)."""
-
-from __future__ import annotations
-
-from typing import get_args
-
-import pytest
-
-from codex.synthesis import Lead, LeadKind, LeadStatus, Provenance
-
-
-def test_provenance_requires_bibkey_and_locator() -> None:
-    """Provenance has two required positional fields."""
-    p = Provenance(bibkey="Smith2020", locator="chunk 7")
-    assert p.bibkey == "Smith2020"
-    assert p.locator == "chunk 7"
-
-
-def test_provenance_missing_field_raises() -> None:
-    """Provenance without locator must fail to construct (TypeError)."""
-    with pytest.raises(TypeError):
-        Provenance(bibkey="Smith2020")  # type: ignore[call-arg]
-
-
-def test_lead_required_fields() -> None:
-    """Lead requires id, kind, title, body, provenance, confidence."""
-    lead = Lead(
-        id="L-0001",
-        kind="connection",
-        title="A connection",
-        body="Some body text.",
-        provenance=[Provenance(bibkey="X2020", locator="chunk 1")],
-        confidence=0.8,
-    )
-    assert lead.id == "L-0001"
-    assert lead.kind == "connection"
-    assert lead.status == "unverified"  # default
-    assert lead.suggested_validation == ""  # default
-    assert lead.claims == []  # default factory
-
-
-def test_lead_kind_literals_exact() -> None:
-    """Lead kinds are exactly: connection, gap, improvement, conjecture."""
-    assert set(get_args(LeadKind)) == {"connection", "gap", "improvement", "conjecture"}
-
-
-def test_lead_status_literals_exact() -> None:
-    """Lead statuses are exactly: unverified, spiking, go, no-go."""
-    assert set(get_args(LeadStatus)) == {"unverified", "spiking", "go", "no-go"}
-
-
-def test_lead_provenance_can_be_multiple() -> None:
-    """A Lead may carry multiple provenance pointers."""
-    provs = [
-        Provenance(bibkey="A2020", locator="chunk 1"),
-        Provenance(bibkey="B2021", locator="chunk 7"),
-    ]
-    lead = Lead(
-        id="L-0002",
-        kind="gap",
-        title="t",
-        body="b",
-        provenance=provs,
-        confidence=0.5,
-    )
-    assert len(lead.provenance) == 2
-
-
-def test_lead_missing_provenance_field_raises() -> None:
-    """Constructing Lead without provenance is a TypeError."""
-    with pytest.raises(TypeError):
-        Lead(  # type: ignore[call-arg]
-            id="L-0003",
-            kind="connection",
-            title="x",
-            body="y",
-            confidence=0.5,
-        )
-
-
-def test_lead_empty_provenance_raises() -> None:
-    """Constructing Lead with empty provenance list raises ValueError."""
-    with pytest.raises(ValueError, match="at least one Provenance"):
-        Lead(
-            id="L-0004",
-            kind="connection",
-            title="x",
-            body="y",
-            provenance=[],
-            confidence=0.5,
-        )
-
-
-def test_lead_id_format_helper() -> None:
-    """_make_lead_id produces a kind-prefixed, zero-padded L-<K>-XXXX string."""
-    from codex.synthesis import _make_lead_id
-
-    assert _make_lead_id(1, "connection") == "L-C-0001"
-    assert _make_lead_id(42, "gap") == "L-G-0042"
-    assert _make_lead_id(9999, "improvement") == "L-I-9999"
-    assert _make_lead_id(1, "conjecture") == "L-X-0001"
-
-
-def test_lead_id_namespaced_per_kind_no_collision() -> None:
-    """Regression for audit C-11: each stage numbers from 1, so without a kind
-    prefix connection/gap/improvement all collide on L-0001 and overwrite one
-    another in grounded/. The prefix must keep same-seq ids distinct.
-    """
-    from codex.synthesis import _make_lead_id
-
-    ids = {_make_lead_id(1, k) for k in ("connection", "gap", "improvement", "conjecture")}
-    assert len(ids) == 4, f"same-seq ids must stay distinct across kinds, got {ids}"