feat(crossref): reference extraction as third citation fallback (R-B)

Add crossref.fetch_references(doi), which parses message.reference[].DOI into Citations (DOI-less book/unstructured refs are skipped — they cannot join the graph). Wire it as the third leg of the ingest citation fallback chain: OpenAlex-empty -> S2 -> Crossref, for DOI papers only (Crossref's works endpoint is DOI-keyed). Cited DOIs are normalized to the canonical bare lower-case form via _norm_cited_id.

As a fallback it fires only when OpenAlex and S2 both return no references, so it adds a forward-looking third source without changing the already-covered corpus.

Tests: fetch_references unit tests (DOI edges / unstructured skipped / no refs / 404) and ingest tests for both fallback directions. Full suite 361 passed; ruff + mypy clean. Live-validated: 33/40/24 refs for Springer/ACM/Wiley DOIs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 21:32:46 +02:00
parent 1516684bbb
commit 82cea2a33b
5 changed files with 202 additions and 4 deletions

View File

@@ -2,10 +2,11 @@
Provides:
- fetch_abstract: retrieve a work's abstract (JATS-stripped) by DOI.
- fetch_references: retrieve a work's reference list (DOI edges) by DOI.
Crossref is a *third* metadata source after OpenAlex and Semantic Scholar
(DQ-2 / roadmap R-B). Requests use the Polite Pool (mailto query parameter)
and are retried on 429/5xx with exponential back-off via tenacity.
(DQ-2 abstracts / roadmap R-B references). Requests use the Polite Pool (mailto
query parameter) and are retried on 429/5xx with exponential back-off via tenacity.
"""
from __future__ import annotations
@@ -18,6 +19,7 @@ import httpx
from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential
from codex.config import get_settings
from codex.models import Citation
logger = logging.getLogger(__name__)
@@ -89,3 +91,36 @@ def fetch_abstract(doi: str) -> str | None:
raise
message: dict[str, Any] = response.json().get("message", {})
return _strip_jats(message.get("abstract"))
def fetch_references(doi: str) -> list[Citation]:
"""Fetch a work's reference list from Crossref by DOI (roadmap R-B).
Crossref carries publisher-deposited reference lists in ``message.reference``.
Each entry that cites a DOI-registered work exposes a bare ``DOI`` field;
only those become citation-graph edges (book / older references carry just
``unstructured`` text with no resolvable id and are skipped). ``citing_id``
is the queried *doi* — the caller rewrites it to the canonical ``papers.id``
and normalises the cited DOIs (mirrors the Semantic Scholar reference path).
Parameters
----------
doi:
A bare DOI (``"10.1007/s00454-019-00132-8"``).
Returns
-------
list[Citation]
One Citation per reference that carries a DOI. Empty on 404 or when no
references are deposited for the work.
"""
url = f"{_BASE}/works/{doi}"
try:
response = _get(url)
except httpx.HTTPStatusError as exc:
if exc.response.status_code == 404:
return []
raise
message: dict[str, Any] = response.json().get("message", {})
references: list[dict[str, Any]] = message.get("reference") or []
return [Citation(citing_id=doi, cited_id=ref["DOI"]) for ref in references if ref.get("DOI")]