feat(crossref): reference extraction as third citation fallback (R-B)
Add crossref.fetch_references(doi), which parses message.reference[].DOI into Citations (DOI-less book/unstructured refs are skipped — they cannot join the graph). Wire it as the third leg of the ingest citation fallback chain: OpenAlex-empty -> S2 -> Crossref, for DOI papers only (Crossref's works endpoint is DOI-keyed). Cited DOIs are normalized to the canonical bare lower-case form via _norm_cited_id. As a fallback it fires only when OpenAlex and S2 both return no references, so it adds a forward-looking third source without changing the already-covered corpus. Tests: fetch_references unit tests (DOI edges / unstructured skipped / no refs / 404) and ingest tests for both fallback directions. Full suite 361 passed; ruff + mypy clean. Live-validated: 33/40/24 refs for Springer/ACM/Wiley DOIs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -2,10 +2,11 @@
|
||||
|
||||
Provides:
|
||||
- fetch_abstract: retrieve a work's abstract (JATS-stripped) by DOI.
|
||||
- fetch_references: retrieve a work's reference list (DOI edges) by DOI.
|
||||
|
||||
Crossref is a *third* metadata source after OpenAlex and Semantic Scholar
|
||||
(DQ-2 / roadmap R-B). Requests use the Polite Pool (mailto query parameter)
|
||||
and are retried on 429/5xx with exponential back-off via tenacity.
|
||||
(DQ-2 abstracts / roadmap R-B references). Requests use the Polite Pool (mailto
|
||||
query parameter) and are retried on 429/5xx with exponential back-off via tenacity.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -18,6 +19,7 @@ import httpx
|
||||
from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential
|
||||
|
||||
from codex.config import get_settings
|
||||
from codex.models import Citation
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -89,3 +91,36 @@ def fetch_abstract(doi: str) -> str | None:
|
||||
raise
|
||||
message: dict[str, Any] = response.json().get("message", {})
|
||||
return _strip_jats(message.get("abstract"))
|
||||
|
||||
|
||||
def fetch_references(doi: str) -> list[Citation]:
|
||||
"""Fetch a work's reference list from Crossref by DOI (roadmap R-B).
|
||||
|
||||
Crossref carries publisher-deposited reference lists in ``message.reference``.
|
||||
Each entry that cites a DOI-registered work exposes a bare ``DOI`` field;
|
||||
only those become citation-graph edges (book / older references carry just
|
||||
``unstructured`` text with no resolvable id and are skipped). ``citing_id``
|
||||
is the queried *doi* — the caller rewrites it to the canonical ``papers.id``
|
||||
and normalises the cited DOIs (mirrors the Semantic Scholar reference path).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
doi:
|
||||
A bare DOI (``"10.1007/s00454-019-00132-8"``).
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[Citation]
|
||||
One Citation per reference that carries a DOI. Empty on 404 or when no
|
||||
references are deposited for the work.
|
||||
"""
|
||||
url = f"{_BASE}/works/{doi}"
|
||||
try:
|
||||
response = _get(url)
|
||||
except httpx.HTTPStatusError as exc:
|
||||
if exc.response.status_code == 404:
|
||||
return []
|
||||
raise
|
||||
message: dict[str, Any] = response.json().get("message", {})
|
||||
references: list[dict[str, Any]] = message.get("reference") or []
|
||||
return [Citation(citing_id=doi, cited_id=ref["DOI"]) for ref in references if ref.get("DOI")]
|
||||
|
||||
Reference in New Issue
Block a user