feat(grobid): run GROBID on-demand (restart "no") to free ~3.5GB when idle

GROBID is only needed during reference extraction (ra_grobid_backfill.py),
not for normal corpus operations (search, embeddings, wiki). Idle it still
holds ~3.5GB on the 8GB Jetson, so keep it stopped by default.

- infra/docker-compose.yml: grobid restart unless-stopped -> "no" (no auto-start
  on boot; db keeps unless-stopped). Start/stop papers-grobid around an R-A run.
- docs/infra/grobid-jetson.md: add "Run on-demand" section with the start/stop
  workflow, and record the refextract head-to-head evaluation on the Jetson that
  led to keeping GROBID: far worse DOI recall on this math corpus (lutz thesis
  101 usable IDs vs refextract 2) and slower per PDF despite a smaller footprint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 21:36:04 +02:00
parent 82cea2a33b
commit bf90c6cd57
2 changed files with 35 additions and 2 deletions

View File

@@ -104,7 +104,36 @@ docker compose logs -f grobid # wait for "Started ... in N seconds" (mode
The `deploy.resources.limits.memory: 4g` cap in the compose file is honored by The `deploy.resources.limits.memory: 4g` cap in the compose file is honored by
Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves
headroom alongside Postgres. headroom alongside Postgres. The `restart: "no"` policy means GROBID does **not**
auto-start on boot — see "Run on-demand" below.
---
## Run on-demand (stop when idle)
GROBID is only needed **during reference extraction** (`scripts/ra_grobid_backfill.py`);
nothing else in the stack uses it. Idle it still holds ~3.5 GB, so on the 8 GB Jetson
keep it stopped and start it only for an R-A run:
```bash
ssh alfred@192.168.178.103 'docker start papers-grobid' # ~30s model load
curl -s --retry 30 --retry-delay 2 --retry-connrefused \
http://192.168.178.103:8070/api/isalive # wait for: true
# ... run R-A from the Mac ...
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write
ssh alfred@192.168.178.103 'docker stop papers-grobid' # reclaim ~3.5 GB
```
First-time deploy uses `docker compose up -d grobid` (above); afterwards the
container persists, so `docker start/stop papers-grobid` is all you need.
> Why on-demand rather than a lighter tool: refextract was evaluated head-to-head
> on the Jetson and rejected — far worse DOI recall on this math corpus (lutz
> thesis: GROBID **101** usable IDs vs refextract **2**), and it was *slower* per
> PDF despite a smaller footprint. GROBID stays; running it on-demand removes its
> only real downside (idle RAM).
--- ---

View File

@@ -44,7 +44,11 @@ services:
init: true # reap zombie children (GROBID docs recommend --init) init: true # reap zombie children (GROBID docs recommend --init)
ulimits: ulimits:
core: 0 # disable core dumps (GROBID docs: --ulimit core=0) core: 0 # disable core dumps (GROBID docs: --ulimit core=0)
restart: unless-stopped # On-demand only: GROBID is needed solely during reference extraction
# (scripts/ra_grobid_backfill.py), not for normal corpus operations — start it
# for an R-A run, stop it after to reclaim ~3.5GB. "no" = no auto-start on boot.
# See docs/infra/grobid-jetson.md ("Run on-demand").
restart: "no"
# GROBID is RAM-hungry; cap usage on the constrained Jetson (Orin Nano, 8GB # GROBID is RAM-hungry; cap usage on the constrained Jetson (Orin Nano, 8GB
# RAM shared with Postgres). 3GB suffices for references; 4g leaves headroom. # RAM shared with Postgres). 3GB suffices for references; 4g leaves headroom.
# Honored by `docker compose up` under Compose v2 (no swarm needed). # Honored by `docker compose up` under Compose v2 (no swarm needed).