feat(grobid): R-A reference backfill + native arm64 GROBID on Jetson
Enable roadmap R-A (GROBID reference extraction) end-to-end and run it against local Jetson infra instead of a Mac/Rosetta emulation. Backfill: - Add scripts/ra_grobid_backfill.py: references-only, idempotent citation backfill (dry-run default). Deliberately not ingest_paper(source_path=pdf), which re-OCRs and replaces the DQ-3-clean .txt chunks; this touches only the citations table (ON CONFLICT DO NOTHING). 7 tests. - Make extract_references timeout configurable (large theses exceed the 60s default, especially against a slower GROBID). Jetson infra: - Bump grobid/grobid 0.8.2 -> 0.9.0-crf. The -crf tag is the only GROBID variant published as an arm64 multi-arch manifest; every 0.8.x tag and the full deep-learning image are amd64-only, which is why GROBID never ran on the aarch64 Jetson. Enable the 4g memory cap + init/ulimits per GROBID docs. - Add docs/infra/grobid-jetson.md runbook: arm64 image rationale, the two host prerequisites (docker-group membership + the Compose v2 CLI plugin, both missing on the Jetson), service-scoped deploy, and the GET /api/isalive check. Verified live 2026-06-17: native arm64 pull, isalive=true, end-to-end extract_references on lutz-2024-thesis.pdf = 116 refs (101 with DOI/arXiv). docs(audit): mark R-A core done (citing coverage 27/29 -> 29/29; edges 920 -> 1022). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -28,20 +28,30 @@ services:
|
||||
restart: unless-stopped
|
||||
|
||||
grobid:
|
||||
# CRF-only image is sufficient for reference extraction and is lighter than
|
||||
# the full deeplearning variant. Check https://hub.docker.com/r/grobid/grobid
|
||||
# for the latest 0.8.x tag before deploying.
|
||||
image: grobid/grobid:0.8.2
|
||||
# CRF-only image: sufficient for reference extraction (/api/processReferences)
|
||||
# and ~500MB vs ~10GB for the full deep-learning variant.
|
||||
#
|
||||
# The `-crf` tag is also the ONLY GROBID variant published as a multi-arch
|
||||
# manifest that includes linux/arm64 — the full image (grobid/grobid:0.9.0)
|
||||
# and every 0.8.x tag are amd64-only. So this is what lets GROBID run
|
||||
# *natively* on the aarch64 Jetson (no slow/RAM-heavy qemu emulation). arm64
|
||||
# builds start at 0.9.0-crf; there is no arm64 0.8.x.
|
||||
# Deploy + docker-permission fix: docs/infra/grobid-jetson.md
|
||||
image: grobid/grobid:0.9.0-crf
|
||||
container_name: papers-grobid
|
||||
ports:
|
||||
- "8070:8070"
|
||||
init: true # reap zombie children (GROBID docs recommend --init)
|
||||
ulimits:
|
||||
core: 0 # disable core dumps (GROBID docs: --ulimit core=0)
|
||||
restart: unless-stopped
|
||||
# GROBID is RAM-hungry; uncomment to cap usage on constrained hardware
|
||||
# (e.g. Nvidia Jetson):
|
||||
# deploy:
|
||||
# resources:
|
||||
# limits:
|
||||
# memory: 4g
|
||||
# GROBID is RAM-hungry; cap usage on the constrained Jetson (Orin Nano, 8GB
|
||||
# RAM shared with Postgres). 3GB suffices for references; 4g leaves headroom.
|
||||
# Honored by `docker compose up` under Compose v2 (no swarm needed).
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4g
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
|
||||
Reference in New Issue
Block a user