feat(ingest): supplement citations from Semantic Scholar when OpenAlex is empty (DQ-1)
12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses
without a parsed reference list (verified referenced_works_count=0 server-side
for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit.
- ingest.py: when OpenAlex returns no references, fall back to a Semantic
Scholar reference supplement (_s2_reference_supplement); citing_id rewritten
to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant
discarded S2 probe in the OpenAlex-404 arXiv branch.
- semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs
responses (.get("data", []) returns None when the key is present-but-null).
- tests: regression test for the OpenAlex-empty -> S2 path.
- Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge
papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce
theses remain (no references in any source).
- docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a
free-source acquisition roadmap (R-A..R-E).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -87,7 +87,11 @@ def fetch_references(paper_id: str) -> list[Citation]:
|
||||
raise
|
||||
|
||||
data = response.json()
|
||||
raw_refs: list[dict[str, Any]] = data.get("data", [])
|
||||
# S2 returns HTTP 200 with ``{"data": null}`` for a known paper that has no
|
||||
# parsed reference list (e.g. some theses). ``.get("data", [])`` would yield
|
||||
# the explicit ``None`` (the default only applies when the key is absent),
|
||||
# so iterate over a guaranteed list instead.
|
||||
raw_refs: list[dict[str, Any]] = data.get("data") or []
|
||||
citations: list[Citation] = []
|
||||
for entry in raw_refs:
|
||||
cited_paper: dict[str, Any] = entry.get("citedPaper", {})
|
||||
|
||||
Reference in New Issue
Block a user