fix(openalex): canonicalize arXiv DOI to bare arXiv id (DQ-5 arXiv half)
OpenAlex returns arXiv works under the arXiv DOI (10.48550/arXiv.<id>); _normalize_doi (DQ-5) left it as 10.48550/arxiv.<id>, but the corpus keys arXiv papers on the BARE id (2305.10988 / math/0603097). So re-ingesting an arXiv paper by its bare id missed INSERT ... ON CONFLICT (id) and tripped papers_openalex_id_key — the arXiv half of DQ-5, explicitly deferred there and surfaced again driving the R-C .tex re-ingest. Add _canonical_id() (normalize DOI, then strip the 10.48550/arxiv. prefix) and use it in _map_paper. Regression tests (modern + legacy arXiv DOI, regular DOI unchanged, fetch_paper bare id). Confirmed live: re-ingesting 2305.10988 now UPDATEs in place. Full suite 370 passed; ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -105,6 +105,27 @@ def _normalize_doi(doi: str) -> str:
|
||||
return s
|
||||
|
||||
|
||||
# arXiv works are registered under a DOI of the form 10.48550/arXiv.<id>.
|
||||
_ARXIV_DOI_PREFIX = "10.48550/arxiv."
|
||||
|
||||
|
||||
def _canonical_id(doi: str) -> str:
|
||||
"""Return the canonical ``papers.id`` for an OpenAlex ``doi`` value.
|
||||
|
||||
Builds on :func:`_normalize_doi` (bare, lower-cased DOI), then strips the
|
||||
arXiv DOI prefix ``10.48550/arxiv.`` so an arXiv work's id is the BARE arXiv
|
||||
id (``2305.10988`` / ``math/0603097``) — the form the corpus is keyed on —
|
||||
not the arXiv DOI. Without this, re-ingesting an arXiv paper by its bare id
|
||||
misses ``INSERT … ON CONFLICT (id)`` and trips ``papers_openalex_id_key``.
|
||||
This is the arXiv half of DQ-5 (the DOI half was fixed there; this was
|
||||
explicitly deferred and surfaced again driving the R-C .tex re-ingest).
|
||||
"""
|
||||
normalized = _normalize_doi(doi)
|
||||
if normalized.startswith(_ARXIV_DOI_PREFIX):
|
||||
return normalized[len(_ARXIV_DOI_PREFIX) :]
|
||||
return normalized
|
||||
|
||||
|
||||
def _map_paper(data: dict[str, Any]) -> Paper:
|
||||
authors: list[str] = [
|
||||
a.get("author", {}).get("display_name") or "" for a in data.get("authorships", [])
|
||||
@@ -112,7 +133,7 @@ def _map_paper(data: dict[str, Any]) -> Paper:
|
||||
abstract = _reconstruct_abstract(data.get("abstract_inverted_index"))
|
||||
doi = data.get("doi")
|
||||
return Paper(
|
||||
id=_normalize_doi(doi) if doi else (data.get("id") or ""),
|
||||
id=_canonical_id(doi) if doi else (data.get("id") or ""),
|
||||
title=data.get("title") or "",
|
||||
openalex_id=data.get("id"),
|
||||
authors=authors,
|
||||
|
||||
Reference in New Issue
Block a user