fix(ingest): canonicalise papers.id to the caller id (audit C-7, T-3)

openalex.fetch_paper fills paper.id from OpenAlex's URL-form doi/id
(e.g. https://doi.org/10.48550/arxiv.math/0603097), and ingest upserted that as
papers.id. Live DB confirmed 35/36 ids were URL-form — breaking bare-id lookups
(graph related / discover citing / search by id) and leaving cited_id matching
to rely solely on openalex_id.

ingest now sets paper.id to the caller-supplied id (the arXiv id / DOI the CLI
and ingest scripts use); OpenAlex's work id is retained as openalex_id and in
paper_identifiers. Regression test: fetch_paper returns a URL-form id, papers.id
upsert uses the bare caller id.

T-3: the OpenAlex test fixture used a bare 'doi' (not the URL form the API
emits) and never asserted paper.id, which masked this. Fixture now uses the URL
form and the test pins fetch_paper's mapping.

NOTE: existing rows keep their URL-form ids until re-ingested — run ingest_all.sh
to migrate the 36-paper corpus (chosen migration path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-15 12:06:53 +02:00
parent 6193f5630d
commit c36e7c6ee7
3 changed files with 45 additions and 1 deletions

View File

@@ -98,6 +98,14 @@ def ingest_paper(
if paper is None:
raise ValueError(f"Paper not found: {paper_id}")
# Canonical identity is the caller-supplied id — what the CLI / ingest scripts
# use and what papers.id is contracted to hold (arXiv id or DOI). OpenAlex's
# fetch_paper fills paper.id with the URL-form doi/id (e.g.
# "https://doi.org/10.48550/arxiv.math/0603097"), which broke bare-id lookups
# and cross-citation matching (audit C-7). The OpenAlex work id is retained
# separately as openalex_id and in the paper_identifiers alias table.
paper.id = paper_id.strip()
# Auto-generate bibkey when source has none (D-04).
if paper.bibkey is None:
paper.bibkey = _make_bibkey(paper)