feat(crossref): reference extraction as third citation fallback (R-B)

Add crossref.fetch_references(doi), which parses message.reference[].DOI into Citations (DOI-less book/unstructured refs are skipped — they cannot join the graph). Wire it as the third leg of the ingest citation fallback chain: OpenAlex-empty -> S2 -> Crossref, for DOI papers only (Crossref's works endpoint is DOI-keyed). Cited DOIs are normalized to the canonical bare lower-case form via _norm_cited_id.

As a fallback it fires only when OpenAlex and S2 both return no references, so it adds a forward-looking third source without changing the already-covered corpus.

Tests: fetch_references unit tests (DOI edges / unstructured skipped / no refs / 404) and ingest tests for both fallback directions. Full suite 361 passed; ruff + mypy clean. Live-validated: 33/40/24 refs for Springer/ACM/Wiley DOIs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 21:32:46 +02:00
parent 1516684bbb
commit 82cea2a33b
5 changed files with 202 additions and 4 deletions

View File

@@ -382,7 +382,35 @@ is self-contained so a cold session can pick it up.
- **Acceptance:** the 2 theses gain references; published papers gain
GROBID-parsed refs beyond OpenAlex/S2 coverage; citing coverage 27/29 → 29/29.
### R-B — Add Crossref as a third citation + abstract source · **MED lever, MED effort**
### R-B — Add Crossref as a third citation + abstract source · **DONE 2026-06-17**
**Resolution (what was done):**
- **`crossref.fetch_references(doi)`** added (mirrors `semanticscholar.fetch_references`):
GETs `/works/{doi}`, returns one `Citation` per `message.reference[]` entry that
carries a registered `DOI` (book/`unstructured`-only refs are skipped — they
can't join the graph). `citing_id` is the queried DOI; the ingest supplement
rewrites it to `paper.id`.
- **Wired as the third leg of the citation fallback chain** in `ingest.py`:
OpenAlex-empty → S2 → **Crossref** (`_crossref_reference_supplement`, DOI papers
only — Crossref's works endpoint is DOI-keyed). cited DOIs pass through
`_norm_cited_id` (bare, lower-cased). Network/parse failures degrade to [].
- **Tests:** `fetch_references` unit tests (DOI edges / `unstructured` skipped / no
`reference` key / 404) + ingest tests asserting the Crossref leg fires when
OpenAlex+S2 are empty (cited DOI normalised) **and** does *not* fire when OpenAlex
has citations. Suite **361 green**, ruff/mypy clean. The DQ-2 abstract half
(`fetch_abstract`) already shipped earlier.
- **Live-validated:** `fetch_references` against the real API returned 33 / 40 / 24
DOI-bearing refs for `10.1007/s00454-019-00132-8` / `10.1145/3132705` /
`10.1111/cgf.13931` (Springer / ACM / Wiley).
- **No live corpus change (by design):** as a *fallback*, Crossref fires only when
OpenAlex **and** S2 are both empty. Post-DQ-1/R-A the only such papers were the 2
`depositonce` theses, which Crossref doesn't index either — so R-B is a
forward-looking third leg for future DOI ingests, not a backfill. (Harvesting
Crossref's *unique* refs for already-covered papers would need an always-merged
supplement instead — a deliberate scope choice not taken, per the doc's "fallback
chain" spec.) **Files:** `codex/sources/crossref.py`, `codex/ingest.py`, + tests.
**Original plan (for context):**
- **What:** new `codex.sources.crossref` client hitting Crossref REST
`/works/{doi}` for the `reference` array and `abstract`.
- **Why:** Crossref carries **publisher-deposited references** for many DOIs (free,