codex-py/README.md

# codex — Personal Paper Knowledge Base

A self-hostable tool for managing scientific papers: ingest PDFs and arXiv
sources, build a citation graph, and link C++ implementations back to the
papers that describe them.

---

## Quick start

```bash
# 1. Start Postgres (pgvector) + GROBID
docker compose -f infra/docker-compose.yml up -d

# 2. Copy and edit environment variables
cp .env.example .env
$EDITOR .env

# 3. Install Python dependencies (requires uv)
uv sync

# 4. Apply the database schema (idempotent — safe to re-run after upgrades)
#    Needs a role that can run DDL. If DATABASE_URL is a least-privilege app
#    role, point MIGRATION_DATABASE_URL at an owner/superuser connection first:
#      MIGRATION_DATABASE_URL=postgresql://postgres:...@host:5432/papers
uv run codex migrate
```

---

## Environment variables

| Variable | Default | Description |
|---|---|---|
| `DATABASE_URL` | `postgresql://researcher:change_me@localhost:5432/papers` | libpq connection string (app role; DML) |
| `MIGRATION_DATABASE_URL` | *(falls back to `DATABASE_URL`)* | Privileged connection used by `codex migrate` for schema DDL (owner/superuser) |
| `GROBID_URL` | `http://localhost:8070` | GROBID HTTP API base URL |
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Local Ollama endpoint (optional) |
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | sentence-transformers model name |
| `EMBEDDING_DIM` | `1024` | Embedding vector dimension (must match model) |
| `OPENALEX_MAILTO` | *(empty)* | E-mail for OpenAlex Polite Pool (required for automated use) |

See `.env.example` for a documented template.

---

## Three-layer data model

All data lives in a single Postgres instance with the pgvector extension.

### Layer 1 — Semantics (`papers`, `chunks`)

Papers are stored with metadata and an abstract-level dense embedding
(BGE-M3, 1024 dimensions). Full-text is split into `chunks`, each with its
own dense embedding and a Postgres full-text (GIN) index.  
Hybrid search combines nearest-neighbour vector search with keyword (FTS)
retrieval for robust handling of exact mathematical terminology.

### Layer 2 — Citations (`citations`)

Directed edges in the citation graph.  The `cited_id` column has **no
foreign-key constraint** on purpose: edges pointing to papers that have not
yet been ingested are kept as-is.  Those "dangling" targets are your
**discovery leads** — papers frequently cited by your collection that you
have not yet read.

```sql
-- Top discovery leads
SELECT cited_id, count(*) AS pull
FROM   citations
WHERE  cited_id NOT IN (SELECT id FROM papers)
GROUP  BY cited_id
ORDER  BY pull DESC
LIMIT  20;
```

### Layer 3 — Provenance (`code_links`)

Maps C++ symbols (qualified names or `file.cpp:line` references) to the
papers they implement.  The workflow:

1. Tag C++ source with `@cite <bibkey>` in Doxygen comments.
2. Run `codex provenance scan --lib-path <path>` to scan and resolve tags.
3. Run `codex provenance bib <out.bib>` to generate a `.bib` file
   containing **only the cited subset** of your collection.

The exported `.bib` is a derived view of the master catalogue — regenerate
it at any time; it is not the source of truth.

---

## CLI reference (coming in F-07)

```
codex ingest <id>               # ingest one paper by arXiv ID or DOI
codex ingest-file <ids.txt>     # bulk ingest from a file of IDs
codex search "<query>" [--hybrid]
codex discover leads
codex provenance scan --lib-path <path>
codex provenance bib <out.bib>
codex ask "<question>"          # optional LLM Q&A via Ollama
```

---

## MCP Server (F-14)

codex exposes its KB as read-only MCP tools for LLM clients (Claude Code, Claude Desktop, etc.).

### Start (stdio)

```bash
# directly
python -m codex.mcp_server

# or via the installed script
codex-mcp
```

### Client registration

**Claude Code** — add to `.mcp.json` in your project root:

```json
{
  "mcpServers": {
    "codex": {
      "command": "uv",
      "args": ["run", "codex-mcp"],
      "cwd": "/path/to/codex-py"
    }
  }
}
```

**Claude Desktop** — add to `~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "codex": {
      "command": "uv",
      "args": ["--directory", "/path/to/codex-py", "run", "codex-mcp"]
    }
  }
}
```

### HTTP transport (optional, config-gated)

```bash
MCP_TRANSPORT=http MCP_AUTH_TOKEN=<secret> codex-mcp
```

HTTP transport requires `MCP_AUTH_TOKEN` — the server refuses to start without it.
Requests must carry `Authorization: Bearer <token>`.

### Available tools (read-only)

| Tool | Description |
|---|---|
| `search` | Chunk-level hybrid search (dense BGE-M3) |
| `ask` | RAG answer (returns "not available" until implemented) |
| `wiki_read` | Compiled concept page (F-12) |
| `wiki_list` | All concept pages + freshness |
| `discover_leads` | Dangling-citation leads |
| `provenance_verify` | Scan C++ sources for @cite tags |
| `synthesis_browse` | Leads/conjectures from leads/ (F-13) |

---

## Development

```bash
uv run ruff check . && uv run ruff format --check .   # lint
uv run mypy codex/                                     # type-check
uv run pytest                                          # tests
```