user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(grobid): run GROBID on-demand (restart "no") to free ~3.5GB when idle

Browse Source 
GROBID is only needed during reference extraction (ra_grobid_backfill.py),
not for normal corpus operations (search, embeddings, wiki). Idle it still
holds ~3.5GB on the 8GB Jetson, so keep it stopped by default.

- infra/docker-compose.yml: grobid restart unless-stopped -> "no" (no auto-start
  on boot; db keeps unless-stopped). Start/stop papers-grobid around an R-A run.
- docs/infra/grobid-jetson.md: add "Run on-demand" section with the start/stop
  workflow, and record the refextract head-to-head evaluation on the Jetson that
  led to keeping GROBID: far worse DOI recall on this math corpus (lutz thesis
  101 usable IDs vs refextract 2) and slower per PDF despite a smaller footprint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 21:36:04 +02:00
parent 82cea2a33b
commit bf90c6cd57
2 changed files with 35 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
docs/infra/grobid-jetson.md
View File

@@ -104,7 +104,36 @@ docker compose logs -f grobid # wait for "Started ... in N seconds" (mode
The `deploy.resources.limits.memory: 4g` cap in the compose file is honored by
Compose v2 here (no swarm needed). 3 GB suffices for references; 4 GB leaves
headroom alongside Postgres.
headroom alongside Postgres. The `restart: "no"` policy means GROBID does **not**
auto-start on boot — see "Run on-demand" below.
---
## Run on-demand (stop when idle)
GROBID is only needed **during reference extraction** (`scripts/ra_grobid_backfill.py`);
nothing else in the stack uses it. Idle it still holds ~3.5 GB, so on the 8 GB Jetson
keep it stopped and start it only for an R-A run:
```bash
ssh alfred@192.168.178.103 'docker start papers-grobid' # ~30s model load
curl -s --retry 30 --retry-delay 2 --retry-connrefused \
http://192.168.178.103:8070/api/isalive # wait for: true
# ... run R-A from the Mac ...
PYTHONPATH=. python scripts/ra_grobid_backfill.py --write
ssh alfred@192.168.178.103 'docker stop papers-grobid' # reclaim ~3.5 GB
```
First-time deploy uses `docker compose up -d grobid` (above); afterwards the
container persists, so `docker start/stop papers-grobid` is all you need.
> Why on-demand rather than a lighter tool: refextract was evaluated head-to-head
> on the Jetson and rejected — far worse DOI recall on this math corpus (lutz
> thesis: GROBID **101** usable IDs vs refextract **2**), and it was *slower* per
> PDF despite a smaller footprint. GROBID stays; running it on-demand removes its
> only real downside (idle RAM).
---