feat(grobid): R-A reference backfill + native arm64 GROBID on Jetson
Enable roadmap R-A (GROBID reference extraction) end-to-end and run it against local Jetson infra instead of a Mac/Rosetta emulation. Backfill: - Add scripts/ra_grobid_backfill.py: references-only, idempotent citation backfill (dry-run default). Deliberately not ingest_paper(source_path=pdf), which re-OCRs and replaces the DQ-3-clean .txt chunks; this touches only the citations table (ON CONFLICT DO NOTHING). 7 tests. - Make extract_references timeout configurable (large theses exceed the 60s default, especially against a slower GROBID). Jetson infra: - Bump grobid/grobid 0.8.2 -> 0.9.0-crf. The -crf tag is the only GROBID variant published as an arm64 multi-arch manifest; every 0.8.x tag and the full deep-learning image are amd64-only, which is why GROBID never ran on the aarch64 Jetson. Enable the 4g memory cap + init/ulimits per GROBID docs. - Add docs/infra/grobid-jetson.md runbook: arm64 image rationale, the two host prerequisites (docker-group membership + the Compose v2 CLI plugin, both missing on the Jetson), service-scoped deploy, and the GET /api/isalive check. Verified live 2026-06-17: native arm64 pull, isalive=true, end-to-end extract_references on lutz-2024-thesis.pdf = 116 refs (101 with DOI/arXiv). docs(audit): mark R-A core done (citing coverage 27/29 -> 29/29; edges 920 -> 1022). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
163
docs/infra/grobid-jetson.md
Normal file
163
docs/infra/grobid-jetson.md
Normal file
@@ -0,0 +1,163 @@
|
||||
# GROBID on the Jetson (aarch64) — deploy runbook
|
||||
|
||||
**Status:** ACTIVE
|
||||
**Host:** `alfred@192.168.178.103` — NVIDIA Jetson Orin Nano (aarch64, 8 GB RAM, 8 GB swap, 6 cores)
|
||||
**Service:** GROBID reference extraction for roadmap **R-A** (`scripts/ra_grobid_backfill.py`)
|
||||
**Last updated:** 2026-06-17
|
||||
**Verified:** 2026-06-17 — `grobid/grobid:0.9.0-crf` pulled as native `arm64`, `/api/isalive`→`true`,
|
||||
end-to-end `extract_references` on `lutz-2024-thesis.pdf` → 116 refs (101 with DOI/arXiv) in ~36s.
|
||||
|
||||
---
|
||||
|
||||
## Why this image (`grobid/grobid:0.9.0-crf`)
|
||||
|
||||
GROBID had never come up on the Jetson because `infra/docker-compose.yml` pinned
|
||||
`grobid/grobid:0.8.2`, and **every `0.8.x` tag (and the full `0.9.0`) is published
|
||||
amd64-only**. A Docker tag resolves to a multi-arch *manifest list*; on aarch64
|
||||
the `0.8.2` list has no `linux/arm64` entry, so there is simply nothing to pull or
|
||||
run (hence `curl localhost:8070` → connection refused, nothing listening).
|
||||
|
||||
The fix is to select a tag whose manifest list actually contains a `linux/arm64`
|
||||
build:
|
||||
|
||||
| Candidate | arm64? | Size | Notes |
|
||||
|-------------------------------|:------:|--------:|----------------------------------------|
|
||||
| `grobid/grobid:0.8.2` | ✗ | ~9.5 GB | full DL image, amd64-only (old pin) |
|
||||
| `grobid/grobid:0.9.0` / `-full`| ✗ | ~10 GB | full DL image, amd64-only |
|
||||
| **`grobid/grobid:0.9.0-crf`** | **✓** | ~500 MB | **CRF-only, official, multi-arch** ← |
|
||||
| `lfoppiano/grobid:0.9.0-crf` | ✓ | ~500 MB | same build, community namespace (alt.) |
|
||||
|
||||
We use **`grobid/grobid:0.9.0-crf`**:
|
||||
|
||||
- **CRF-only** is sufficient for reference extraction (`/api/processReferences`)
|
||||
and is ~500 MB vs ~10 GB for the deep-learning image.
|
||||
- **Native arm64** — no qemu emulation (too slow / RAM-heavy on a Jetson).
|
||||
- **Official namespace**, **pinned version** (not `latest-crf`, which is a moving
|
||||
target). arm64 builds only exist from `0.9.0-crf` onward — there is no arm64
|
||||
`0.8.x`, so this is a forced (but compatible) bump from the old `0.8.2` pin. The
|
||||
TEI that `codex/parsing/grobid.py` parses (`biblStruct`, `persName`,
|
||||
`idno[@type='DOI']`, …) is unchanged across the bump.
|
||||
|
||||
> Caveat from upstream: the arm64 image is documented as "tested only on macOS,
|
||||
> not linux/arm64". The container is the *same* `linux/arm64` ELF either way, so
|
||||
> bare-metal Jetson works; the end-to-end check below is what de-risks it.
|
||||
|
||||
---
|
||||
|
||||
## One-time prerequisites
|
||||
|
||||
### 1. Docker permission for `alfred`
|
||||
|
||||
The SSH user `alfred` is in the `sudo` group but **not** the `docker` group, so
|
||||
`docker …` fails with `permission denied … /var/run/docker.sock`. Grant access
|
||||
once (requires the sudo password):
|
||||
|
||||
```bash
|
||||
ssh alfred@192.168.178.103
|
||||
sudo usermod -aG docker alfred # add to docker group (persistent)
|
||||
```
|
||||
|
||||
Then **start a fresh login** so the new group takes effect (group membership is
|
||||
only re-evaluated at login — an existing session/`ssh` won't see it; a *new*
|
||||
`ssh` connection will):
|
||||
|
||||
```bash
|
||||
exit && ssh alfred@192.168.178.103
|
||||
docker ps # should now work without sudo
|
||||
```
|
||||
|
||||
(Alternatively run all `docker` commands below under `sudo`, but group membership
|
||||
is cleaner and lets `ra_grobid_backfill.py` / curl checks be driven over plain SSH.)
|
||||
|
||||
### 2. Docker Compose v2 plugin (arm64)
|
||||
|
||||
This host has Docker Engine but **not** the Compose v2 CLI plugin (on Linux they
|
||||
are separate packages; only Docker Desktop bundles Compose). Without it
|
||||
`docker compose …` fails with `docker: unknown command: docker compose`. Install
|
||||
the arm64 plugin into the user-local plugin dir (no sudo needed):
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.docker/cli-plugins
|
||||
curl -sSL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-aarch64 \
|
||||
-o ~/.docker/cli-plugins/docker-compose
|
||||
chmod +x ~/.docker/cli-plugins/docker-compose
|
||||
docker compose version # confirm it resolves
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deploy
|
||||
|
||||
GROBID is **service-scoped** on purpose: Postgres (`papers-db`) is already running
|
||||
on this host, so we bring up only the `grobid` service and never `docker compose up`
|
||||
the whole stack (that would try to start a second `papers-db`).
|
||||
|
||||
```bash
|
||||
# copy this repo's compose file to the Jetson (first time only)
|
||||
scp infra/docker-compose.yml infra/schema.sql alfred@192.168.178.103:~/codex-infra/
|
||||
|
||||
ssh alfred@192.168.178.103
|
||||
cd ~/codex-infra
|
||||
docker compose up -d grobid # pulls grobid/grobid:0.9.0-crf (arm64), starts papers-grobid
|
||||
docker compose logs -f grobid # wait for "Started ... in N seconds" (model load ~20–40s)
|
||||
```
|
||||
|
||||
The `deploy.resources.limits.memory: 4g` cap in the compose file is honored by
|
||||
Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves
|
||||
headroom alongside Postgres.
|
||||
|
||||
---
|
||||
|
||||
## Verify `GET /api/isalive` → `true`
|
||||
|
||||
```bash
|
||||
# on the Jetson
|
||||
curl -s http://localhost:8070/api/isalive # => true
|
||||
|
||||
# from the Mac (LAN) — same value the corpus' GROBID_URL points at
|
||||
curl -s http://192.168.178.103:8070/api/isalive # => true
|
||||
```
|
||||
|
||||
Anything other than `true` (timeout / connection refused / `false`) means the
|
||||
container isn't up — check `docker compose logs grobid` and that nothing else
|
||||
holds port 8070.
|
||||
|
||||
---
|
||||
|
||||
## End-to-end reference-extraction check + running R-A
|
||||
|
||||
With `GROBID_URL=http://192.168.178.103:8070` (already set in `.env.jetson-ingest`):
|
||||
|
||||
```bash
|
||||
# one PDF, references only — should print N>0 parsed references
|
||||
PYTHONPATH=. python - <<'PY'
|
||||
from codex.parsing.grobid import extract_references
|
||||
refs = extract_references("/Users/tarikmoussa/Desktop/ConformalLabpp/papers/lutz-2024-thesis.pdf",
|
||||
grobid_url="http://192.168.178.103:8070")
|
||||
print(f"{len(refs)} references; first DOI/arXiv:",
|
||||
next(((r['doi'], r['arxiv_id']) for r in refs if r['doi'] or r['arxiv_id']), None))
|
||||
PY
|
||||
```
|
||||
|
||||
Then run the references-only backfill (DB tunnel up — see
|
||||
`docs/audit/DATA-QUALITY-2026-06-15.md`):
|
||||
|
||||
```bash
|
||||
ssh -N -L 5433:localhost:5432 alfred@192.168.178.103 & # DB tunnel
|
||||
cp .env.jetson-ingest .env
|
||||
PYTHONPATH=. python scripts/ra_grobid_backfill.py --zero-edge-only # dry-run first
|
||||
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write # apply
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Container killed mid-request / exit 137** — OOM. The CRF image needs ~3 GB for
|
||||
references; raise the compose `memory` limit or check Postgres isn't starving the
|
||||
box (`free -h`, `docker stats papers-grobid`).
|
||||
- **`isalive` true but `processReferences` 500s** — usually a `pdfalto` failure on a
|
||||
malformed PDF; confirm with a clean PDF (`lutz-2024-thesis.pdf`) and check
|
||||
`docker compose logs grobid`.
|
||||
- **`permission denied … docker.sock`** — the docker-group prerequisite above wasn't
|
||||
applied, or the SSH session predates it (re-login).
|
||||
Reference in New Issue
Block a user