codex-py/docs/infra/grobid-jetson.md

GROBID on the Jetson (aarch64) — deploy runbook

Status: ACTIVE Host: alfred@jetson.local — NVIDIA Jetson Orin Nano (aarch64, 8 GB RAM, 8 GB swap, 6 cores) Service: GROBID reference extraction for roadmap R-A (scripts/ra_grobid_backfill.py) Last updated: 2026-06-17 Verified: 2026-06-17 — grobid/grobid:0.9.0-crf pulled as native arm64, /api/isalive→true, end-to-end extract_references on lutz-2024-thesis.pdf → 116 refs (101 with DOI/arXiv) in ~36s.

---

Why this image (grobid/grobid:0.9.0-crf)

GROBID had never come up on the Jetson because infra/docker-compose.yml pinned grobid/grobid:0.8.2, and every 0.8.x tag (and the full 0.9.0) is published amd64-only. A Docker tag resolves to a multi-arch manifest list; on aarch64 the 0.8.2 list has no linux/arm64 entry, so there is simply nothing to pull or run (hence curl localhost:8070 → connection refused, nothing listening).

The fix is to select a tag whose manifest list actually contains a linux/arm64 build:

Candidate	arm64?	Size	Notes
grobid/grobid:0.8.2	✗	~9.5 GB	full DL image, amd64-only (old pin)
grobid/grobid:0.9.0 / -full	✗	~10 GB	full DL image, amd64-only
grobid/grobid:0.9.0-crf	✓	~500 MB	CRF-only, official, multi-arch ←
lfoppiano/grobid:0.9.0-crf	✓	~500 MB	same build, community namespace (alt.)

We use grobid/grobid:0.9.0-crf:

• CRF-only is sufficient for reference extraction (/api/processReferences) and is ~500 MB vs ~10 GB for the deep-learning image.
• Native arm64 — no qemu emulation (too slow / RAM-heavy on a Jetson).
• Official namespace, pinned version (not latest-crf, which is a moving target). arm64 builds only exist from 0.9.0-crf onward — there is no arm64 0.8.x, so this is a forced (but compatible) bump from the old 0.8.2 pin. The TEI that codex/parsing/grobid.py parses (biblStruct, persName, idno[@type='DOI'], …) is unchanged across the bump.

Caveat from upstream: the arm64 image is documented as "tested only on macOS, not linux/arm64". The container is the same linux/arm64 ELF either way, so bare-metal Jetson works; the end-to-end check below is what de-risks it.

---

One-time prerequisites

1. Docker permission for alfred

The SSH user alfred is in the sudo group but not the docker group, so docker … fails with permission denied … /var/run/docker.sock. Grant access once (requires the sudo password):

ssh alfred@jetson.local
sudo usermod -aG docker alfred      # add to docker group (persistent)

Then start a fresh login so the new group takes effect (group membership is only re-evaluated at login — an existing session/ssh won't see it; a new ssh connection will):

exit && ssh alfred@jetson.local
docker ps                            # should now work without sudo

(Alternatively run all docker commands below under sudo, but group membership is cleaner and lets ra_grobid_backfill.py / curl checks be driven over plain SSH.)

2. Docker Compose v2 plugin (arm64)

This host has Docker Engine but not the Compose v2 CLI plugin (on Linux they are separate packages; only Docker Desktop bundles Compose). Without it docker compose … fails with docker: unknown command: docker compose. Install the arm64 plugin into the user-local plugin dir (no sudo needed):

mkdir -p ~/.docker/cli-plugins
curl -sSL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-aarch64 \
  -o ~/.docker/cli-plugins/docker-compose
chmod +x ~/.docker/cli-plugins/docker-compose
docker compose version               # confirm it resolves

---

Deploy

GROBID is service-scoped on purpose: Postgres (papers-db) is already running on this host, so we bring up only the grobid service and never docker compose up the whole stack (that would try to start a second papers-db).

# copy this repo's compose file to the Jetson (first time only)
scp infra/docker-compose.yml infra/schema.sql alfred@jetson.local:~/codex-infra/

ssh alfred@jetson.local
cd ~/codex-infra
docker compose up -d grobid          # pulls grobid/grobid:0.9.0-crf (arm64), starts papers-grobid
docker compose logs -f grobid        # wait for "Started ... in N seconds" (model load ~20–40s)

The deploy.resources.limits.memory: 4g cap in the compose file is honored by Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves headroom alongside Postgres. The restart: "no" policy means GROBID does not auto-start on boot — see "Run on-demand" below.

---

Run on-demand (stop when idle)

GROBID is only needed during reference extraction (scripts/ra_grobid_backfill.py); nothing else in the stack uses it. Idle it still holds ~3.5 GB, so on the 8 GB Jetson keep it stopped and start it only for an R-A run:

ssh alfred@jetson.local 'docker start papers-grobid'        # ~30s model load
curl -s --retry 30 --retry-delay 2 --retry-connrefused \
  http://jetson.local:8070/api/isalive                      # wait for: true

# ... run R-A from the Mac ...
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write

ssh alfred@jetson.local 'docker stop papers-grobid'         # reclaim ~3.5 GB

First-time deploy uses docker compose up -d grobid (above); afterwards the container persists, so docker start/stop papers-grobid is all you need.

Why on-demand rather than a lighter tool: refextract was evaluated head-to-head on the Jetson and rejected — far worse DOI recall on this math corpus (lutz thesis: GROBID 101 usable IDs vs refextract 2), and it was slower per PDF despite a smaller footprint. GROBID stays; running it on-demand removes its only real downside (idle RAM).

---

Verify GET /api/isalive → true

# on the Jetson
curl -s http://localhost:8070/api/isalive          # => true

# from the Mac (LAN) — same value the corpus' GROBID_URL points at
curl -s http://jetson.local:8070/api/isalive     # => true

Anything other than true (timeout / connection refused / false) means the container isn't up — check docker compose logs grobid and that nothing else holds port 8070.

---

End-to-end reference-extraction check + running R-A

With GROBID_URL=http://jetson.local:8070 (already set in .env.jetson-ingest):

# one PDF, references only — should print N>0 parsed references
PYTHONPATH=. python - <<'PY'
from codex.parsing.grobid import extract_references
refs = extract_references("/Users/tarikmoussa/Desktop/ConformalLabpp/papers/lutz-2024-thesis.pdf",
                          grobid_url="http://jetson.local:8070")
print(f"{len(refs)} references; first DOI/arXiv:",
      next(((r['doi'], r['arxiv_id']) for r in refs if r['doi'] or r['arxiv_id']), None))
PY

Then run the references-only backfill (DB tunnel up — see docs/audit/DATA-QUALITY-2026-06-15.md):

ssh -N -L 5433:localhost:5432 alfred@jetson.local &     # DB tunnel
cp .env.jetson-ingest .env
PYTHONPATH=. python scripts/ra_grobid_backfill.py --zero-edge-only   # dry-run first
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write            # apply

---

Troubleshooting

• Container killed mid-request / exit 137 — OOM. The CRF image needs ~3 GB for references; raise the compose memory limit or check Postgres isn't starving the box (free -h, docker stats papers-grobid).
• isalive true but processReferences 500s — usually a pdfalto failure on a malformed PDF; confirm with a clean PDF (lutz-2024-thesis.pdf) and check docker compose logs grobid.
• permission denied … docker.sock — the docker-group prerequisite above wasn't applied, or the SSH session predates it (re-login).