codex-py/docs/infra/grobid-jetson.md

# GROBID on the Jetson (aarch64) — deploy runbook

**Status:** ACTIVE
**Host:** `alfred@192.168.178.103` — NVIDIA Jetson Orin Nano (aarch64, 8 GB RAM, 8 GB swap, 6 cores)
**Service:** GROBID reference extraction for roadmap **R-A** (`scripts/ra_grobid_backfill.py`)
**Last updated:** 2026-06-17
**Verified:** 2026-06-17 — `grobid/grobid:0.9.0-crf` pulled as native `arm64`, `/api/isalive`→`true`,
end-to-end `extract_references` on `lutz-2024-thesis.pdf` → 116 refs (101 with DOI/arXiv) in ~36s.

---

## Why this image (`grobid/grobid:0.9.0-crf`)

GROBID had never come up on the Jetson because `infra/docker-compose.yml` pinned
`grobid/grobid:0.8.2`, and **every `0.8.x` tag (and the full `0.9.0`) is published
amd64-only**. A Docker tag resolves to a multi-arch *manifest list*; on aarch64
the `0.8.2` list has no `linux/arm64` entry, so there is simply nothing to pull or
run (hence `curl localhost:8070` → connection refused, nothing listening).

The fix is to select a tag whose manifest list actually contains a `linux/arm64`
build:

| Candidate                     | arm64? | Size    | Notes                                  |
|-------------------------------|:------:|--------:|----------------------------------------|
| `grobid/grobid:0.8.2`         | ✗      | ~9.5 GB | full DL image, amd64-only (old pin)    |
| `grobid/grobid:0.9.0` / `-full`| ✗     | ~10 GB  | full DL image, amd64-only              |
| **`grobid/grobid:0.9.0-crf`** | **✓**  | ~500 MB | **CRF-only, official, multi-arch** ←   |
| `lfoppiano/grobid:0.9.0-crf`  | ✓      | ~500 MB | same build, community namespace (alt.) |

We use **`grobid/grobid:0.9.0-crf`**:

- **CRF-only** is sufficient for reference extraction (`/api/processReferences`)
  and is ~500 MB vs ~10 GB for the deep-learning image.
- **Native arm64** — no qemu emulation (too slow / RAM-heavy on a Jetson).
- **Official namespace**, **pinned version** (not `latest-crf`, which is a moving
  target). arm64 builds only exist from `0.9.0-crf` onward — there is no arm64
  `0.8.x`, so this is a forced (but compatible) bump from the old `0.8.2` pin. The
  TEI that `codex/parsing/grobid.py` parses (`biblStruct`, `persName`,
  `idno[@type='DOI']`, …) is unchanged across the bump.

> Caveat from upstream: the arm64 image is documented as "tested only on macOS,
> not linux/arm64". The container is the *same* `linux/arm64` ELF either way, so
> bare-metal Jetson works; the end-to-end check below is what de-risks it.

---

## One-time prerequisites

### 1. Docker permission for `alfred`

The SSH user `alfred` is in the `sudo` group but **not** the `docker` group, so
`docker …` fails with `permission denied … /var/run/docker.sock`. Grant access
once (requires the sudo password):

```bash
ssh alfred@192.168.178.103
sudo usermod -aG docker alfred      # add to docker group (persistent)
```

Then **start a fresh login** so the new group takes effect (group membership is
only re-evaluated at login — an existing session/`ssh` won't see it; a *new*
`ssh` connection will):

```bash
exit && ssh alfred@192.168.178.103
docker ps                            # should now work without sudo
```

(Alternatively run all `docker` commands below under `sudo`, but group membership
is cleaner and lets `ra_grobid_backfill.py` / curl checks be driven over plain SSH.)

### 2. Docker Compose v2 plugin (arm64)

This host has Docker Engine but **not** the Compose v2 CLI plugin (on Linux they
are separate packages; only Docker Desktop bundles Compose). Without it
`docker compose …` fails with `docker: unknown command: docker compose`. Install
the arm64 plugin into the user-local plugin dir (no sudo needed):

```bash
mkdir -p ~/.docker/cli-plugins
curl -sSL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-aarch64 \
  -o ~/.docker/cli-plugins/docker-compose
chmod +x ~/.docker/cli-plugins/docker-compose
docker compose version               # confirm it resolves
```

---

## Deploy

GROBID is **service-scoped** on purpose: Postgres (`papers-db`) is already running
on this host, so we bring up only the `grobid` service and never `docker compose up`
the whole stack (that would try to start a second `papers-db`).

```bash
# copy this repo's compose file to the Jetson (first time only)
scp infra/docker-compose.yml infra/schema.sql alfred@192.168.178.103:~/codex-infra/

ssh alfred@192.168.178.103
cd ~/codex-infra
docker compose up -d grobid          # pulls grobid/grobid:0.9.0-crf (arm64), starts papers-grobid
docker compose logs -f grobid        # wait for "Started ... in N seconds" (model load ~20–40s)
```

The `deploy.resources.limits.memory: 4g` cap in the compose file is honored by
Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves
headroom alongside Postgres. The `restart: "no"` policy means GROBID does **not**
auto-start on boot — see "Run on-demand" below.

---

## Run on-demand (stop when idle)

GROBID is only needed **during reference extraction** (`scripts/ra_grobid_backfill.py`);
nothing else in the stack uses it. Idle it still holds ~3.5 GB, so on the 8 GB Jetson
keep it stopped and start it only for an R-A run:

```bash
ssh alfred@192.168.178.103 'docker start papers-grobid'        # ~30s model load
curl -s --retry 30 --retry-delay 2 --retry-connrefused \
  http://192.168.178.103:8070/api/isalive                      # wait for: true

# ... run R-A from the Mac ...
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write

ssh alfred@192.168.178.103 'docker stop papers-grobid'         # reclaim ~3.5 GB
```

First-time deploy uses `docker compose up -d grobid` (above); afterwards the
container persists, so `docker start/stop papers-grobid` is all you need.

> Why on-demand rather than a lighter tool: refextract was evaluated head-to-head
> on the Jetson and rejected — far worse DOI recall on this math corpus (lutz
> thesis: GROBID **101** usable IDs vs refextract **2**), and it was *slower* per
> PDF despite a smaller footprint. GROBID stays; running it on-demand removes its
> only real downside (idle RAM).

---

## Verify `GET /api/isalive` → `true`

```bash
# on the Jetson
curl -s http://localhost:8070/api/isalive          # => true

# from the Mac (LAN) — same value the corpus' GROBID_URL points at
curl -s http://192.168.178.103:8070/api/isalive     # => true
```

Anything other than `true` (timeout / connection refused / `false`) means the
container isn't up — check `docker compose logs grobid` and that nothing else
holds port 8070.

---

## End-to-end reference-extraction check + running R-A

With `GROBID_URL=http://192.168.178.103:8070` (already set in `.env.jetson-ingest`):

```bash
# one PDF, references only — should print N>0 parsed references
PYTHONPATH=. python - <<'PY'
from codex.parsing.grobid import extract_references
refs = extract_references("/Users/tarikmoussa/Desktop/ConformalLabpp/papers/lutz-2024-thesis.pdf",
                          grobid_url="http://192.168.178.103:8070")
print(f"{len(refs)} references; first DOI/arXiv:",
      next(((r['doi'], r['arxiv_id']) for r in refs if r['doi'] or r['arxiv_id']), None))
PY
```

Then run the references-only backfill (DB tunnel up — see
`docs/audit/DATA-QUALITY-2026-06-15.md`):

```bash
ssh -N -L 5433:localhost:5432 alfred@192.168.178.103 &     # DB tunnel
cp .env.jetson-ingest .env
PYTHONPATH=. python scripts/ra_grobid_backfill.py --zero-edge-only   # dry-run first
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write            # apply
```

---

## Troubleshooting

- **Container killed mid-request / exit 137** — OOM. The CRF image needs ~3 GB for
  references; raise the compose `memory` limit or check Postgres isn't starving the
  box (`free -h`, `docker stats papers-grobid`).
- **`isalive` true but `processReferences` 500s** — usually a `pdfalto` failure on a
  malformed PDF; confirm with a clean PDF (`lutz-2024-thesis.pdf`) and check
  `docker compose logs grobid`.
- **`permission denied … docker.sock`** — the docker-group prerequisite above wasn't
  applied, or the SSH session predates it (re-login).