codex — Personal Paper Knowledge Base

A self-hostable tool for managing scientific papers: ingest PDFs and arXiv sources, build a citation graph, and link C++ implementations back to the papers that describe them.

---

Quick start

# 1. Start Postgres (pgvector) + GROBID
docker compose -f infra/docker-compose.yml up -d

# 2. Copy and edit environment variables
cp .env.example .env
$EDITOR .env

# 3. Install Python dependencies (requires uv)
uv sync

# 4. Apply the database schema (first run only)
uv run python -c "
from codex.db import get_conn, apply_schema
with get_conn() as conn:
    apply_schema(conn)
print('Schema applied.')
"

---

Environment variables

Variable	Default	Description
DATABASE_URL	postgresql://researcher:change_me@localhost:5432/papers	libpq connection string
GROBID_URL	http://localhost:8070	GROBID HTTP API base URL
OLLAMA_BASE_URL	http://localhost:11434	Local Ollama endpoint (optional)
EMBEDDING_MODEL	BAAI/bge-m3	sentence-transformers model name
EMBEDDING_DIM	1024	Embedding vector dimension (must match model)
OPENALEX_MAILTO	(empty)	E-mail for OpenAlex Polite Pool (required for automated use)

See .env.example for a documented template.

---

Three-layer data model

All data lives in a single Postgres instance with the pgvector extension.

Layer 1 — Semantics (papers, chunks)

Papers are stored with metadata and an abstract-level dense embedding (BGE-M3, 1024 dimensions). Full-text is split into chunks, each with its own dense embedding and a Postgres full-text (GIN) index.
Hybrid search combines nearest-neighbour vector search with keyword (FTS) retrieval for robust handling of exact mathematical terminology.

Layer 2 — Citations (citations)

Directed edges in the citation graph. The cited_id column has no foreign-key constraint on purpose: edges pointing to papers that have not yet been ingested are kept as-is. Those "dangling" targets are your discovery leads — papers frequently cited by your collection that you have not yet read.

-- Top discovery leads
SELECT cited_id, count(*) AS pull
FROM   citations
WHERE  cited_id NOT IN (SELECT id FROM papers)
GROUP  BY cited_id
ORDER  BY pull DESC
LIMIT  20;

Layer 3 — Provenance (code_links)

Maps C++ symbols (qualified names or file.cpp:line references) to the papers they implement. The workflow:

1. Tag C++ source with @cite <bibkey> in Doxygen comments.
2. Run codex provenance scan --lib-path <path> to scan and resolve tags.
3. Run codex provenance bib <out.bib> to generate a .bib file containing only the cited subset of your collection.

The exported .bib is a derived view of the master catalogue — regenerate it at any time; it is not the source of truth.

---

CLI reference (coming in F-07)

codex ingest <id>               # ingest one paper by arXiv ID or DOI
codex ingest-file <ids.txt>     # bulk ingest from a file of IDs
codex search "<query>" [--hybrid]
codex discover leads
codex provenance scan --lib-path <path>
codex provenance bib <out.bib>
codex ask "<question>"          # optional LLM Q&A via Ollama

---

MCP Server (F-14)

codex exposes its KB as read-only MCP tools for LLM clients (Claude Code, Claude Desktop, etc.).

Start (stdio)

# directly
python -m codex.mcp_server

# or via the installed script
codex-mcp

Client registration

Claude Code — add to .mcp.json in your project root:

{
  "mcpServers": {
    "codex": {
      "command": "uv",
      "args": ["run", "codex-mcp"],
      "cwd": "/path/to/codex-py"
    }
  }
}

Claude Desktop — add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "codex": {
      "command": "uv",
      "args": ["--directory", "/path/to/codex-py", "run", "codex-mcp"]
    }
  }
}

HTTP transport (optional, config-gated)

MCP_TRANSPORT=http MCP_AUTH_TOKEN=<secret> codex-mcp

HTTP transport requires MCP_AUTH_TOKEN — the server refuses to start without it. Requests must carry Authorization: Bearer <token>.

Available tools (read-only)

Tool	Description
search	Chunk-level hybrid search (dense BGE-M3)
ask	RAG answer (returns "not available" until implemented)
wiki_read	Compiled concept page (F-12)
wiki_list	All concept pages + freshness
discover_leads	Dangling-citation leads
provenance_verify	Scan C++ sources for @cite tags
synthesis_browse	Leads/conjectures from leads/ (F-13)

---

Development

uv run ruff check . && uv run ruff format --check .   # lint
uv run mypy codex/                                     # type-check
uv run pytest                                          # tests