user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(openalex): normalize DOI to bare canonical id so re-ingest is idempotent (DQ-5)

Browse Source 
openalex._map_paper set Paper.id from the raw `doi` field, which OpenAlex
returns as a full URL (https://doi.org/10.x). The canonical papers.id (M-1
migration) is the bare, lower-cased DOI, so re-ingesting a DOI paper missed
ON CONFLICT (id) and tripped the separate papers_openalex_id_key constraint
(UniqueViolation).

Fix: _normalize_doi() strips https://doi.org//http://doi.org//doi: and
lower-cases; applied in the DOI branch of _map_paper only (W-id fallback
untouched). Also un-breaks _s2_id_for/Crossref recovery, which the URL form
silently defeated. Corrected the _SAMPLE_WORK fixture to the URL form (the bare
fixture hid the bug) + added _normalize_doi and re-ingest idempotency regression
tests.

Live verification: re-ingest of 10.1007/s00454-019-00132-8 (previously crashed)
now UPDATEs in place — single row, added_at unchanged, no stray row, citations
45->45. Unblocks roadmap R-A/R-C. Full suite 348 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 00:31:01 +02:00
parent fa43f69dfd
commit ea73b1475c
4 changed files with 181 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
codex/sources/openalex.py
View File

@@ -85,13 +85,34 @@ def _reconstruct_abstract(inverted_index: dict[str, list[int]] | None) -> str |
return " ".join(w for _, w in word_positions)
def _normalize_doi(doi: str) -> str:
"""Normalize an OpenAlex DOI to the canonical bare, lower-cased form.
OpenAlex returns the ``doi`` field as a full URL
(``https://doi.org/10.1007/s00454-019-00132-8``), but the canonical
``papers.id`` produced by the M-1 migration is the BARE, lower-cased DOI
(``10.1007/s00454-019-00132-8``). Stripping the ``https://doi.org/`` /
``doi:`` prefix and lower-casing (DOIs are case-insensitive by spec) makes a
re-ingested DOI paper match its stored row, so the ingest
``INSERT … ON CONFLICT (id)`` updates in place instead of attempting a fresh
INSERT that trips the separate ``papers_openalex_id_key`` unique constraint
(DQ-5). Mirrors the cited-id convention in ``ingest._norm_cited_id``.
"""
s = doi.strip().lower()
for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
if s.startswith(prefix):
return s[len(prefix) :]
return s
def _map_paper(data: dict[str, Any]) -> Paper:
authors: list[str] = [
a.get("author", {}).get("display_name") or "" for a in data.get("authorships", [])
]
abstract = _reconstruct_abstract(data.get("abstract_inverted_index"))
doi = data.get("doi")
return Paper(
id=data.get("doi") or data.get("id") or "",
id=_normalize_doi(doi) if doi else (data.get("id") or ""),
title=data.get("title") or "",
openalex_id=data.get("id"),
authors=authors,