feat(crossref): reference extraction as third citation fallback (R-B)
Add crossref.fetch_references(doi), which parses message.reference[].DOI into Citations (DOI-less book/unstructured refs are skipped — they cannot join the graph). Wire it as the third leg of the ingest citation fallback chain: OpenAlex-empty -> S2 -> Crossref, for DOI papers only (Crossref's works endpoint is DOI-keyed). Cited DOIs are normalized to the canonical bare lower-case form via _norm_cited_id. As a fallback it fires only when OpenAlex and S2 both return no references, so it adds a forward-looking third source without changing the already-covered corpus. Tests: fetch_references unit tests (DOI edges / unstructured skipped / no refs / 404) and ingest tests for both fallback directions. Full suite 361 passed; ruff + mypy clean. Live-validated: 33/40/24 refs for Springer/ACM/Wiley DOIs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -130,6 +130,29 @@ def _s2_reference_supplement(paper: Paper) -> list[Citation]:
|
||||
]
|
||||
|
||||
|
||||
def _crossref_reference_supplement(paper: Paper) -> list[Citation]:
|
||||
"""Fetch a paper's references from Crossref (roadmap R-B supplement).
|
||||
|
||||
Third leg of the citation fallback chain (OpenAlex-empty → S2 → Crossref).
|
||||
Crossref carries publisher-deposited reference lists keyed on the citing
|
||||
work's DOI, so only DOI papers can be looked up (arXiv-only ids → []).
|
||||
``citing_id`` is rewritten to the canonical ``paper.id`` and cited DOIs are
|
||||
case-normalised. Network/parse failures degrade to [] rather than aborting.
|
||||
"""
|
||||
if not paper.id.startswith("10."):
|
||||
return []
|
||||
try:
|
||||
raw = crossref.fetch_references(paper.id)
|
||||
except Exception:
|
||||
logger.warning("Crossref reference supplement failed for %s", paper.id, exc_info=True)
|
||||
return []
|
||||
return [
|
||||
Citation(citing_id=paper.id, cited_id=_norm_cited_id(c.cited_id), context=c.context)
|
||||
for c in raw
|
||||
if c.cited_id
|
||||
]
|
||||
|
||||
|
||||
def ingest_paper(
|
||||
paper_id: str,
|
||||
source_path: str | None = None,
|
||||
@@ -315,7 +338,7 @@ def ingest_paper(
|
||||
chunks_upserted = len(chunk_rows)
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 5. Fetch citations (OpenAlex if openalex_id, else S2)
|
||||
# 5. Fetch citations (fallback chain: OpenAlex → S2 → Crossref)
|
||||
# Insert with ON CONFLICT (citing_id, cited_id) DO NOTHING
|
||||
# ---------------------------------------------------------------
|
||||
api_citations: list[Citation]
|
||||
@@ -333,6 +356,10 @@ def ingest_paper(
|
||||
api_citations = _s2_reference_supplement(paper)
|
||||
else:
|
||||
api_citations = _s2_reference_supplement(paper)
|
||||
# R-B: if OpenAlex and S2 both came back empty, try Crossref's
|
||||
# publisher-deposited references (DOI papers only) as a third leg.
|
||||
if not api_citations:
|
||||
api_citations = _crossref_reference_supplement(paper)
|
||||
|
||||
# Merge API citations and GROBID PDF citations; dedup via set
|
||||
all_citations_set: set[tuple[str, str]] = set()
|
||||
|
||||
Reference in New Issue
Block a user