merge: R-B (Crossref references as third citation fallback)
This commit is contained in:
@@ -382,7 +382,35 @@ is self-contained so a cold session can pick it up.
|
||||
- **Acceptance:** the 2 theses gain references; published papers gain
|
||||
GROBID-parsed refs beyond OpenAlex/S2 coverage; citing coverage 27/29 → 29/29.
|
||||
|
||||
### R-B — Add Crossref as a third citation + abstract source · **MED lever, MED effort**
|
||||
### R-B — Add Crossref as a third citation + abstract source · **DONE 2026-06-17**
|
||||
|
||||
**Resolution (what was done):**
|
||||
- **`crossref.fetch_references(doi)`** added (mirrors `semanticscholar.fetch_references`):
|
||||
GETs `/works/{doi}`, returns one `Citation` per `message.reference[]` entry that
|
||||
carries a registered `DOI` (book/`unstructured`-only refs are skipped — they
|
||||
can't join the graph). `citing_id` is the queried DOI; the ingest supplement
|
||||
rewrites it to `paper.id`.
|
||||
- **Wired as the third leg of the citation fallback chain** in `ingest.py`:
|
||||
OpenAlex-empty → S2 → **Crossref** (`_crossref_reference_supplement`, DOI papers
|
||||
only — Crossref's works endpoint is DOI-keyed). cited DOIs pass through
|
||||
`_norm_cited_id` (bare, lower-cased). Network/parse failures degrade to [].
|
||||
- **Tests:** `fetch_references` unit tests (DOI edges / `unstructured` skipped / no
|
||||
`reference` key / 404) + ingest tests asserting the Crossref leg fires when
|
||||
OpenAlex+S2 are empty (cited DOI normalised) **and** does *not* fire when OpenAlex
|
||||
has citations. Suite **361 green**, ruff/mypy clean. The DQ-2 abstract half
|
||||
(`fetch_abstract`) already shipped earlier.
|
||||
- **Live-validated:** `fetch_references` against the real API returned 33 / 40 / 24
|
||||
DOI-bearing refs for `10.1007/s00454-019-00132-8` / `10.1145/3132705` /
|
||||
`10.1111/cgf.13931` (Springer / ACM / Wiley).
|
||||
- **No live corpus change (by design):** as a *fallback*, Crossref fires only when
|
||||
OpenAlex **and** S2 are both empty. Post-DQ-1/R-A the only such papers were the 2
|
||||
`depositonce` theses, which Crossref doesn't index either — so R-B is a
|
||||
forward-looking third leg for future DOI ingests, not a backfill. (Harvesting
|
||||
Crossref's *unique* refs for already-covered papers would need an always-merged
|
||||
supplement instead — a deliberate scope choice not taken, per the doc's "fallback
|
||||
chain" spec.) **Files:** `codex/sources/crossref.py`, `codex/ingest.py`, + tests.
|
||||
|
||||
**Original plan (for context):**
|
||||
- **What:** new `codex.sources.crossref` client hitting Crossref REST
|
||||
`/works/{doi}` for the `reference` array and `abstract`.
|
||||
- **Why:** Crossref carries **publisher-deposited references** for many DOIs (free,
|
||||
|
||||
@@ -104,7 +104,36 @@ docker compose logs -f grobid # wait for "Started ... in N seconds" (mode
|
||||
|
||||
The `deploy.resources.limits.memory: 4g` cap in the compose file is honored by
|
||||
Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves
|
||||
headroom alongside Postgres.
|
||||
headroom alongside Postgres. The `restart: "no"` policy means GROBID does **not**
|
||||
auto-start on boot — see "Run on-demand" below.
|
||||
|
||||
---
|
||||
|
||||
## Run on-demand (stop when idle)
|
||||
|
||||
GROBID is only needed **during reference extraction** (`scripts/ra_grobid_backfill.py`);
|
||||
nothing else in the stack uses it. Idle it still holds ~3.5 GB, so on the 8 GB Jetson
|
||||
keep it stopped and start it only for an R-A run:
|
||||
|
||||
```bash
|
||||
ssh alfred@192.168.178.103 'docker start papers-grobid' # ~30s model load
|
||||
curl -s --retry 30 --retry-delay 2 --retry-connrefused \
|
||||
http://192.168.178.103:8070/api/isalive # wait for: true
|
||||
|
||||
# ... run R-A from the Mac ...
|
||||
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write
|
||||
|
||||
ssh alfred@192.168.178.103 'docker stop papers-grobid' # reclaim ~3.5 GB
|
||||
```
|
||||
|
||||
First-time deploy uses `docker compose up -d grobid` (above); afterwards the
|
||||
container persists, so `docker start/stop papers-grobid` is all you need.
|
||||
|
||||
> Why on-demand rather than a lighter tool: refextract was evaluated head-to-head
|
||||
> on the Jetson and rejected — far worse DOI recall on this math corpus (lutz
|
||||
> thesis: GROBID **101** usable IDs vs refextract **2**), and it was *slower* per
|
||||
> PDF despite a smaller footprint. GROBID stays; running it on-demand removes its
|
||||
> only real downside (idle RAM).
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user