feat(grobid): R-A reference backfill + native arm64 GROBID on Jetson
Enable roadmap R-A (GROBID reference extraction) end-to-end and run it against local Jetson infra instead of a Mac/Rosetta emulation. Backfill: - Add scripts/ra_grobid_backfill.py: references-only, idempotent citation backfill (dry-run default). Deliberately not ingest_paper(source_path=pdf), which re-OCRs and replaces the DQ-3-clean .txt chunks; this touches only the citations table (ON CONFLICT DO NOTHING). 7 tests. - Make extract_references timeout configurable (large theses exceed the 60s default, especially against a slower GROBID). Jetson infra: - Bump grobid/grobid 0.8.2 -> 0.9.0-crf. The -crf tag is the only GROBID variant published as an arm64 multi-arch manifest; every 0.8.x tag and the full deep-learning image are amd64-only, which is why GROBID never ran on the aarch64 Jetson. Enable the 4g memory cap + init/ulimits per GROBID docs. - Add docs/infra/grobid-jetson.md runbook: arm64 image rationale, the two host prerequisites (docker-group membership + the Compose v2 CLI plugin, both missing on the Jetson), service-scoped deploy, and the GET /api/isalive check. Verified live 2026-06-17: native arm64 pull, isalive=true, end-to-end extract_references on lutz-2024-thesis.pdf = 116 refs (101 with DOI/arXiv). docs(audit): mark R-A core done (citing coverage 27/29 -> 29/29; edges 920 -> 1022). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -338,7 +338,33 @@ so the quality lever is *combining more free sources*, not buying paywall access
|
||||
Paywall/uni-login was explicitly considered and **deferred** (see R-E). Each item
|
||||
is self-contained so a cold session can pick it up.
|
||||
|
||||
### R-A — Run GROBID reference extraction on arXiv PDFs · **HIGH lever, LOW-MED effort**
|
||||
### R-A — Run GROBID reference extraction on PDFs · **CORE DONE 2026-06-17** (full sweep optional)
|
||||
|
||||
**Resolution (what was done) — citing coverage 27/29 → 29/29 (100%); edges 920 → 1022:**
|
||||
- **References-only backfill, not full re-ingest.** Added
|
||||
[scripts/ra_grobid_backfill.py](../../scripts/ra_grobid_backfill.py): per paper,
|
||||
run `grobid.extract_references(pdf)`, normalize cited-ids to the canonical bare
|
||||
form (DOIs de-URL'd + lower-cased; `arXiv:` stripped), INSERT into `citations`
|
||||
(`ON CONFLICT DO NOTHING`). Deliberately avoids `ingest_paper(source_path=pdf)`,
|
||||
whose PDF branch re-runs Nougat + `DELETE FROM chunks` and would overwrite the
|
||||
DQ-3-verified-clean `.txt` chunks (and needs Nougat). Idempotent; dry-run by
|
||||
default; 7 tests; ruff/mypy clean.
|
||||
- **GROBID provisioning:** the pinned `grobid/grobid:0.8.2` is amd64-only, but the
|
||||
Jetson and the dev Mac are both arm64 (this is why GROBID never ran). Ran the
|
||||
lighter CRF image (`grobid/grobid:0.8.2-crf`) on the Mac under Podman+Rosetta
|
||||
(PDFs are local; citations written over the SSH DB tunnel). Made
|
||||
`extract_references` timeout configurable (theses need >60s emulated). A separate
|
||||
task is provisioning a native arm64 GROBID on the Jetson (`docs/infra/grobid-jetson.md`).
|
||||
- **Both `depositonce` theses closed:** `…depositonce-20357` (lutz-2024) **+96**
|
||||
edges (5 in-corpus); `…depositonce-5415` (sechelmann-2016) **+6** edges — its
|
||||
147 MB/173-page scan blew pdfalto's 120s limit, so the 9 bibliography pages were
|
||||
extracted with PyMuPDF first (77 refs, only 6 with a DOI/arXiv). **No paper has
|
||||
zero out-edges now.**
|
||||
- **Optional remaining (secondary acceptance):** sweep the other 27 already-covered
|
||||
papers for *extra* GROBID refs via `python scripts/ra_grobid_backfill.py --write`
|
||||
(the 2 book PDFs need the same bib-page extraction as sechelmann).
|
||||
|
||||
**Original plan (for context):**
|
||||
- **✓ Unblocked (DQ-5 RESOLVED 2026-06-17):** DOI re-ingest is now idempotent, so
|
||||
the GROBID re-ingest no longer aborts on `papers_openalex_id_key`.
|
||||
- **What:** the whole corpus was ingested from `.txt` (`PAPERS_DIR=…/papers/txt`),
|
||||
|
||||
Reference in New Issue
Block a user