# codex — Personal Paper Knowledge Base

A self-hostable tool for managing scientific papers: ingest PDFs and arXiv
sources, build a citation graph, and link C++ implementations back to the
papers that describe them.

---

## Quick start

```bash
# 1. Start Postgres (pgvector) + GROBID
docker compose -f infra/docker-compose.yml up -d

# 2. Copy and edit environment variables
cp .env.example .env
$EDITOR .env

# 3. Install Python dependencies (requires uv)
uv sync

# 4. Apply the database schema (first run only)
uv run python -c "
from codex.db import get_conn, apply_schema
with get_conn() as conn:
    apply_schema(conn)
print('Schema applied.')
"
```

---

## Environment variables

| Variable | Default | Description |
|---|---|---|
| `DATABASE_URL` | `postgresql://researcher:change_me@localhost:5432/papers` | libpq connection string |
| `GROBID_URL` | `http://localhost:8070` | GROBID HTTP API base URL |
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Local Ollama endpoint (optional) |
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | sentence-transformers model name |
| `EMBEDDING_DIM` | `1024` | Embedding vector dimension (must match model) |
| `OPENALEX_MAILTO` | *(empty)* | E-mail for OpenAlex Polite Pool (required for automated use) |

See `.env.example` for a documented template.

---

## Three-layer data model

All data lives in a single Postgres instance with the pgvector extension.

### Layer 1 — Semantics (`papers`, `chunks`)

Papers are stored with metadata and an abstract-level dense embedding
(BGE-M3, 1024 dimensions). Full-text is split into `chunks`, each with its
own dense embedding and a Postgres full-text (GIN) index.  
Hybrid search combines nearest-neighbour vector search with keyword (FTS)
retrieval for robust handling of exact mathematical terminology.

### Layer 2 — Citations (`citations`)

Directed edges in the citation graph.  The `cited_id` column has **no
foreign-key constraint** on purpose: edges pointing to papers that have not
yet been ingested are kept as-is.  Those "dangling" targets are your
**discovery leads** — papers frequently cited by your collection that you
have not yet read.

```sql
-- Top discovery leads
SELECT cited_id, count(*) AS pull
FROM   citations
WHERE  cited_id NOT IN (SELECT id FROM papers)
GROUP  BY cited_id
ORDER  BY pull DESC
LIMIT  20;
```

### Layer 3 — Provenance (`code_links`)

Maps C++ symbols (qualified names or `file.cpp:line` references) to the
papers they implement.  The workflow:

1. Tag C++ source with `@cite <bibkey>` in Doxygen comments.
2. Run `codex provenance sync --lib-path <path>` to scan and resolve tags.
3. Run `codex provenance export-bib <out.bib>` to generate a `.bib` file
   containing **only the cited subset** of your collection.

The exported `.bib` is a derived view of the master catalogue — regenerate
it at any time; it is not the source of truth.

---

## CLI reference (coming in F-07)

```
codex ingest <id>               # ingest one paper by arXiv ID or DOI
codex ingest-file <ids.txt>     # bulk ingest from a file of IDs
codex search "<query>" [--hybrid]
codex discover leads
codex provenance sync --lib-path <path>
codex provenance export-bib <out.bib>
codex ask "<question>"          # optional LLM Q&A via Ollama
```

---

## Development

```bash
uv run ruff check . && uv run ruff format --check .   # lint
uv run mypy codex/                                     # type-check
uv run pytest                                          # tests
```