M-1: schema.sql could not be re-applied (leading CREATE TABLE/INDEX lacked
IF NOT EXISTS), apply_schema was never called, and the live migration just
failed on a missing chunks.section column. Fixes:
- schema.sql: all CREATE TABLE/INDEX now use IF NOT EXISTS — the whole file is
re-applyable as a no-op.
- codex migrate: new CLI command that applies schema.sql. Connects via
MIGRATION_DATABASE_URL (falls back to DATABASE_URL) and catches
InsufficientPrivilege with guidance — because the app role is DML-only and
core tables are owned by 'postgres' (the privilege dimension found during the
live migration incident).
- config: migration_database_url (optional privileged connection).
- db: apply_schema docstring corrected (idempotency now true) + privilege note.
- T-2: static test that schema.sql is fully idempotent; migrate privilege-error
test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The helper TRUNCATEd papers CASCADE and only then ran ingest_all.sh, which
failed on the missing chunks.section column (audit M-1) — leaving the corpus
wiped. Add a preflight that verifies chunks.section exists BEFORE the
destructive step and aborts early (no TRUNCATE) with the owner-level ALTER to
run, since the app user cannot alter postgres-owned tables.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One-time, guarded migration for the C-7 id-format change: a plain re-ingest
would duplicate every paper (new bare-id row beside the old URL-id PK), so this
wipes (TRUNCATE papers CASCADE) and rebuilds via ingest_all.sh.
Safety: requires the SSH tunnel, prints a BEFORE snapshot, gates the TRUNCATE
behind an explicit 'MIGRATE' confirmation, then prints AFTER verification —
C-7 (url_form_ids should be 0) and C-1 via the real resolver-based
discovery_leads() (ingested papers leaked should be 0). Uses .venv psycopg
(psql is not installed); DATABASE_URL is sourced, never echoed.
Joins PR #13 (Wave 2). Read-only verification SQL validated against the live DB.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes preprint/journal duplicate problem: arXiv papers were ingested
under their preprint OpenAlex ID, but corpus citations reference the
published-version OpenAlex ID → those citations were invisible in the
graph (appeared as dangling).
Changes:
- infra/schema.sql: CREATE TABLE paper_identifiers (paper_id, openalex_id)
with UNIQUE index; seeded from papers.openalex_id on migration
- codex/graph.py: build_citation_graph LEFT JOINs paper_identifiers as
a second resolution path: COALESCE(p.id, pi.paper_id, c.cited_id)
- codex/ingest.py: every ingest inserts openalex_id into paper_identifiers
(ON CONFLICT DO NOTHING) — aliases can be added manually alongside
Live DB: 35 rows seeded + 4 published-version aliases added:
math/0503219 → W2163787581 (DCG 2007)
math/0306167 → W2136126748 (Commun Analysis Geom 2004)
math/0203250 → W1567166970 (Trans AMS 2003)
1005.2698 → W1511400044 (Geom & Topol 2015)
Effect: dangling 574→570; Bobenko+Springborn 2007 now IN-KB rank 9.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
