fix(ingest): canonicalise papers.id to the caller id (audit C-7, T-3)

openalex.fetch_paper fills paper.id from OpenAlex's URL-form doi/id
(e.g. https://doi.org/10.48550/arxiv.math/0603097), and ingest upserted that as
papers.id. Live DB confirmed 35/36 ids were URL-form — breaking bare-id lookups
(graph related / discover citing / search by id) and leaving cited_id matching
to rely solely on openalex_id.

ingest now sets paper.id to the caller-supplied id (the arXiv id / DOI the CLI
and ingest scripts use); OpenAlex's work id is retained as openalex_id and in
paper_identifiers. Regression test: fetch_paper returns a URL-form id, papers.id
upsert uses the bare caller id.

T-3: the OpenAlex test fixture used a bare 'doi' (not the URL form the API
emits) and never asserted paper.id, which masked this. Fixture now uses the URL
form and the test pins fetch_paper's mapping.

NOTE: existing rows keep their URL-form ids until re-ingested — run ingest_all.sh
to migrate the 36-paper corpus (chosen migration path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-15 12:06:53 +02:00
parent 6193f5630d
commit c36e7c6ee7
3 changed files with 45 additions and 1 deletions

View File

@@ -15,7 +15,8 @@ from codex.sources.openalex import _resolve_id
_SAMPLE_WORK = {
"id": "https://openalex.org/W2741809807",
"doi": "10.1145/3592430",
# Real OpenAlex returns the DOI as a full URL, not a bare DOI (audit T-3).
"doi": "https://doi.org/10.1145/3592430",
"title": "Conformal Prediction: A Review",
"publication_year": 2023,
"authorships": [
@@ -90,6 +91,10 @@ def test_fetch_paper_success(monkeypatch: pytest.MonkeyPatch) -> None:
assert paper.authors == ["Alice Smith", "Bob Jones"]
assert paper.abstract == "Conformal prediction is great"
assert paper.openalex_id == "https://openalex.org/W2741809807"
# fetch_paper maps id from the (URL-form) doi; ingest later canonicalises
# papers.id to the caller's bare id (audit C-7). Pin the real mapping here so
# the fixture cannot silently misrepresent OpenAlex again (audit T-3).
assert paper.id == "https://doi.org/10.1145/3592430"
def test_fetch_paper_arxiv_id_prefixed(monkeypatch: pytest.MonkeyPatch) -> None: