fix(openalex): canonicalize arXiv DOI to bare arXiv id (DQ-5 arXiv half)

OpenAlex returns arXiv works under the arXiv DOI (10.48550/arXiv.<id>); _normalize_doi (DQ-5) left it as 10.48550/arxiv.<id>, but the corpus keys arXiv papers on the BARE id (2305.10988 / math/0603097). So re-ingesting an arXiv paper by its bare id missed INSERT ... ON CONFLICT (id) and tripped papers_openalex_id_key — the arXiv half of DQ-5, explicitly deferred there and surfaced again driving the R-C .tex re-ingest.

Add _canonical_id() (normalize DOI, then strip the 10.48550/arxiv. prefix) and use it in _map_paper. Regression tests (modern + legacy arXiv DOI, regular DOI unchanged, fetch_paper bare id). Confirmed live: re-ingesting 2305.10988 now UPDATEs in place. Full suite 370 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-18 04:25:22 +02:00
parent 121b582f46
commit f504ca62db
2 changed files with 59 additions and 2 deletions

View File

@@ -105,6 +105,27 @@ def _normalize_doi(doi: str) -> str:
return s
# arXiv works are registered under a DOI of the form 10.48550/arXiv.<id>.
_ARXIV_DOI_PREFIX = "10.48550/arxiv."
def _canonical_id(doi: str) -> str:
"""Return the canonical ``papers.id`` for an OpenAlex ``doi`` value.
Builds on :func:`_normalize_doi` (bare, lower-cased DOI), then strips the
arXiv DOI prefix ``10.48550/arxiv.`` so an arXiv work's id is the BARE arXiv
id (``2305.10988`` / ``math/0603097``) — the form the corpus is keyed on —
not the arXiv DOI. Without this, re-ingesting an arXiv paper by its bare id
misses ``INSERT … ON CONFLICT (id)`` and trips ``papers_openalex_id_key``.
This is the arXiv half of DQ-5 (the DOI half was fixed there; this was
explicitly deferred and surfaced again driving the R-C .tex re-ingest).
"""
normalized = _normalize_doi(doi)
if normalized.startswith(_ARXIV_DOI_PREFIX):
return normalized[len(_ARXIV_DOI_PREFIX) :]
return normalized
def _map_paper(data: dict[str, Any]) -> Paper:
authors: list[str] = [
a.get("author", {}).get("display_name") or "" for a in data.get("authorships", [])
@@ -112,7 +133,7 @@ def _map_paper(data: dict[str, Any]) -> Paper:
abstract = _reconstruct_abstract(data.get("abstract_inverted_index"))
doi = data.get("doi")
return Paper(
id=_normalize_doi(doi) if doi else (data.get("id") or ""),
id=_canonical_id(doi) if doi else (data.get("id") or ""),
title=data.get("title") or "",
openalex_id=data.get("id"),
authors=authors,