bf90c6cd57
GROBID is only needed during reference extraction (ra_grobid_backfill.py), not for normal corpus operations (search, embeddings, wiki). Idle it still holds ~3.5GB on the 8GB Jetson, so keep it stopped by default. - infra/docker-compose.yml: grobid restart unless-stopped -> "no" (no auto-start on boot; db keeps unless-stopped). Start/stop papers-grobid around an R-A run. - docs/infra/grobid-jetson.md: add "Run on-demand" section with the start/stop workflow, and record the refextract head-to-head evaluation on the Jetson that led to keeping GROBID: far worse DOI recall on this math corpus (lutz thesis 101 usable IDs vs refextract 2) and slower per PDF despite a smaller footprint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
62 lines
2.4 KiB
YAML
62 lines
2.4 KiB
YAML
# Paper knowledge base: Provenance + Citation graph + Semantic search
|
|
# A single Postgres instance (pgvector) covers all three layers.
|
|
# GROBID parses references / structure from PDFs without an arXiv source.
|
|
#
|
|
# Quick start:
|
|
# docker compose up -d
|
|
# psql "${DATABASE_URL}" # see .env.example for default value
|
|
|
|
services:
|
|
db:
|
|
image: pgvector/pgvector:pg17 # official pgvector image
|
|
container_name: papers-db
|
|
environment:
|
|
POSTGRES_DB: papers
|
|
POSTGRES_USER: researcher
|
|
POSTGRES_PASSWORD: change_me # change before production use
|
|
ports:
|
|
- "5432:5432"
|
|
volumes:
|
|
- pgdata:/var/lib/postgresql/data
|
|
# schema.sql is executed automatically on first start:
|
|
- ./schema.sql:/docker-entrypoint-initdb.d/01-schema.sql:ro
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U researcher -d papers"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
restart: unless-stopped
|
|
|
|
grobid:
|
|
# CRF-only image: sufficient for reference extraction (/api/processReferences)
|
|
# and ~500MB vs ~10GB for the full deep-learning variant.
|
|
#
|
|
# The `-crf` tag is also the ONLY GROBID variant published as a multi-arch
|
|
# manifest that includes linux/arm64 — the full image (grobid/grobid:0.9.0)
|
|
# and every 0.8.x tag are amd64-only. So this is what lets GROBID run
|
|
# *natively* on the aarch64 Jetson (no slow/RAM-heavy qemu emulation). arm64
|
|
# builds start at 0.9.0-crf; there is no arm64 0.8.x.
|
|
# Deploy + docker-permission fix: docs/infra/grobid-jetson.md
|
|
image: grobid/grobid:0.9.0-crf
|
|
container_name: papers-grobid
|
|
ports:
|
|
- "8070:8070"
|
|
init: true # reap zombie children (GROBID docs recommend --init)
|
|
ulimits:
|
|
core: 0 # disable core dumps (GROBID docs: --ulimit core=0)
|
|
# On-demand only: GROBID is needed solely during reference extraction
|
|
# (scripts/ra_grobid_backfill.py), not for normal corpus operations — start it
|
|
# for an R-A run, stop it after to reclaim ~3.5GB. "no" = no auto-start on boot.
|
|
# See docs/infra/grobid-jetson.md ("Run on-demand").
|
|
restart: "no"
|
|
# GROBID is RAM-hungry; cap usage on the constrained Jetson (Orin Nano, 8GB
|
|
# RAM shared with Postgres). 3GB suffices for references; 4g leaves headroom.
|
|
# Honored by `docker compose up` under Compose v2 (no swarm needed).
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 4g
|
|
|
|
volumes:
|
|
pgdata:
|