feat(crossref): reference extraction as third citation fallback (R-B)

Add crossref.fetch_references(doi), which parses message.reference[].DOI into Citations (DOI-less book/unstructured refs are skipped — they cannot join the graph). Wire it as the third leg of the ingest citation fallback chain: OpenAlex-empty -> S2 -> Crossref, for DOI papers only (Crossref's works endpoint is DOI-keyed). Cited DOIs are normalized to the canonical bare lower-case form via _norm_cited_id.

As a fallback it fires only when OpenAlex and S2 both return no references, so it adds a forward-looking third source without changing the already-covered corpus.

Tests: fetch_references unit tests (DOI edges / unstructured skipped / no refs / 404) and ingest tests for both fallback directions. Full suite 361 passed; ruff + mypy clean. Live-validated: 33/40/24 refs for Springer/ACM/Wiley DOIs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 21:32:46 +02:00
parent 1516684bbb
commit 82cea2a33b
5 changed files with 202 additions and 4 deletions

View File

@@ -130,6 +130,29 @@ def _s2_reference_supplement(paper: Paper) -> list[Citation]:
]
def _crossref_reference_supplement(paper: Paper) -> list[Citation]:
"""Fetch a paper's references from Crossref (roadmap R-B supplement).
Third leg of the citation fallback chain (OpenAlex-empty → S2 → Crossref).
Crossref carries publisher-deposited reference lists keyed on the citing
work's DOI, so only DOI papers can be looked up (arXiv-only ids → []).
``citing_id`` is rewritten to the canonical ``paper.id`` and cited DOIs are
case-normalised. Network/parse failures degrade to [] rather than aborting.
"""
if not paper.id.startswith("10."):
return []
try:
raw = crossref.fetch_references(paper.id)
except Exception:
logger.warning("Crossref reference supplement failed for %s", paper.id, exc_info=True)
return []
return [
Citation(citing_id=paper.id, cited_id=_norm_cited_id(c.cited_id), context=c.context)
for c in raw
if c.cited_id
]
def ingest_paper(
paper_id: str,
source_path: str | None = None,
@@ -315,7 +338,7 @@ def ingest_paper(
chunks_upserted = len(chunk_rows)
# ---------------------------------------------------------------
# 5. Fetch citations (OpenAlex if openalex_id, else S2)
# 5. Fetch citations (fallback chain: OpenAlex → S2 → Crossref)
# Insert with ON CONFLICT (citing_id, cited_id) DO NOTHING
# ---------------------------------------------------------------
api_citations: list[Citation]
@@ -333,6 +356,10 @@ def ingest_paper(
api_citations = _s2_reference_supplement(paper)
else:
api_citations = _s2_reference_supplement(paper)
# R-B: if OpenAlex and S2 both came back empty, try Crossref's
# publisher-deposited references (DOI papers only) as a third leg.
if not api_citations:
api_citations = _crossref_reference_supplement(paper)
# Merge API citations and GROBID PDF citations; dedup via set
all_citations_set: set[tuple[str, str]] = set()