# Infra TODO — stable long-running ops against the Jetson (tmux + autossh)

**Status:** OPEN (infra). **Filed:** 2026-06-17. **Priority:** MED — it slows live ops
but has a known workaround (short/idempotent ops).

## Problem (observed)

Live data ops run on the **Mac** and reach the Jetson Postgres through an SSH
port-forward tunnel (`ssh -f -N -L 5433:localhost:5432 alfred@192.168.178.103`,
`DATABASE_URL` → `localhost:5433`). During the R-A / R-C / R-F work this tunnel
**dropped repeatedly mid-run**: any operation longer than a few minutes (corpus
re-ingest, the bge-m3 embed loop, the GROBID sweep) had a high chance of failing
partway when the tunnel died, leaving the work half-applied. At one point the Jetson
went **fully offline** (ping 100 % loss, `ssh:22` timeout) mid-operation.

`ingest_paper()` and the backfill scripts open one DB connection for the whole run, so
a tunnel drop raises `OperationalError: connection refused` and aborts — partial state,
manual retry. Adding `ServerAliveInterval` to the tunnel helped only marginally.

## Recommended fix

**1. Run long ops *on the Jetson* inside `tmux` (primary fix).**
The Jetson is where Postgres (and GROBID) live, so running ingest there means the DB is
`localhost` — **no tunnel at all** — and `tmux` keeps the process alive across SSH
disconnects (detach / reattach to monitor). This removes the tunnel from the critical
path for every long-running write.

- [ ] Confirm the `codex` env is deployable on the Jetson (Python 3.12+, deps incl. the
      bge-m3 embedder; the Orin Nano GPU should handle it — GROBID already runs there).
      If not, stand up a venv / container for it.
- [ ] Standard pattern: `ssh alfred@jetson`, then
      `tmux new -s codex` → `set -a; source .env; set +a` →
      `PYTHONPATH=. python scripts/<job>.py --write` → detach (`Ctrl-b d`).
      Reattach later with `tmux attach -t codex`. The job survives the SSH drop.
- [ ] Point the Jetson-side `DATABASE_URL` at `localhost:5432` directly (no `:5433`).
- [ ] Document this as the default for any multi-minute write (re-ingest, sweeps,
      embed loops) in `docs/audit/DATA-QUALITY-2026-06-15.md` "How to reach the data".

**2. `autossh` for when a tunnel *is* needed (secondary).**
For interactive/dev use from the Mac (psql, quick probes, read-only queries), replace
the plain `ssh -f -N -L` with `autossh -M 0 -f -N -o ServerAliveInterval=15 -o
ServerAliveCountMax=3 -L 5433:localhost:5432 alfred@jetson` so a dropped tunnel
auto-reconnects. NB: autossh only restarts the *tunnel* — an in-flight transaction
still fails, so this is for short/interactive use, not long writes (use tmux-on-Jetson
for those).

**3. (Optional, code) make backfill scripts resilient.**
Lower-value once (1)/(2) are in place, but: open a fresh `get_conn()` per paper (not one
for the whole run) and/or wrap the per-item write in a small retry, so a transient drop
costs one item, not the whole batch. The R-A backfill already does per-paper connections;
`rc_tex_reingest.py` opens a connection per paper too — but `ingest_paper` itself holds
one connection for its whole body, which is the unit that fails.

## Why this matters

The chronic flakiness directly cost time this session and left one cosmetic task
unfinished (the R-F accent re-clean — see DATA-QUALITY-2026-06-15.md R-F "Pending"). The
**robust pattern that already works** is short, idempotent operations (per-paper, or a
single quick `UPDATE`); tmux-on-Jetson makes even the long ops safe.

## Acceptance

- A documented, repeatable way to run a full corpus re-ingest that survives an SSH drop
  (tmux-on-Jetson), verified by detaching/reattaching across a disconnect.
- The DATA-QUALITY doc's "How to reach the data" section recommends tmux-on-Jetson for
  long writes and `autossh` for interactive tunnels.