fix(ingest): canonicalise papers.id to the caller id (audit C-7, T-3)

openalex.fetch_paper fills paper.id from OpenAlex's URL-form doi/id
(e.g. https://doi.org/10.48550/arxiv.math/0603097), and ingest upserted that as
papers.id. Live DB confirmed 35/36 ids were URL-form — breaking bare-id lookups
(graph related / discover citing / search by id) and leaving cited_id matching
to rely solely on openalex_id.

ingest now sets paper.id to the caller-supplied id (the arXiv id / DOI the CLI
and ingest scripts use); OpenAlex's work id is retained as openalex_id and in
paper_identifiers. Regression test: fetch_paper returns a URL-form id, papers.id
upsert uses the bare caller id.

T-3: the OpenAlex test fixture used a bare 'doi' (not the URL form the API
emits) and never asserted paper.id, which masked this. Fixture now uses the URL
form and the test pins fetch_paper's mapping.

NOTE: existing rows keep their URL-form ids until re-ingested — run ingest_all.sh
to migrate the 36-paper corpus (chosen migration path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-15 12:06:53 +02:00
parent 6193f5630d
commit c36e7c6ee7
3 changed files with 45 additions and 1 deletions

View File

@@ -106,6 +106,37 @@ def test_ingest_paper_basic() -> None:
)
def test_ingest_canonicalises_paper_id_to_caller_arg() -> None:
"""papers.id is the caller-supplied id, not OpenAlex's URL-form id (audit C-7).
Real OpenAlex returns the doi/id as a full URL; ingest must key the paper on
the bare id the caller passed (matching ingest scripts and CLI lookups), and
keep the OpenAlex work id only as openalex_id.
"""
oa_paper = Paper(
id="https://doi.org/10.48550/arxiv.math/0603097", # URL form, as OpenAlex emits
title="A Test Paper",
openalex_id="https://openalex.org/W4301005794",
authors=["Alice", "Bob"],
year=2008,
abstract="Test abstract.",
)
mock_conn = MagicMock()
with (
patch("codex.ingest.openalex.fetch_paper", return_value=oa_paper),
patch("codex.ingest.openalex.fetch_citations", return_value=[]),
patch("codex.ingest.get_embedder", return_value=_fake_embedder()),
patch("codex.ingest.get_conn", side_effect=_make_conn_cm(mock_conn)),
):
result = ingest_paper("math/0603097")
papers_params = mock_conn.execute.call_args_list[0][0][1]
assert papers_params["id"] == "math/0603097", "papers.id must be the caller's bare id"
assert papers_params["openalex_id"] == "https://openalex.org/W4301005794"
assert result.paper_id == "math/0603097"
# ---------------------------------------------------------------------------
# 2. test_ingest_paper_source_tex
# ---------------------------------------------------------------------------