#1 ingest: normalize the pinned caller id (_norm_cited_id) so a non-bare DOI caller cannot store a URL-form papers.id that defeats DQ-5 idempotency / the startswith(10.) recovery gates. #2 quality.section_label: collapse to a controlled bucket only for an EXACT canonical heading; descriptive titles ('Abstract Nonsense...') keep their real title instead of being mislabelled. #4 ra_grobid_backfill: release the read connection before the slow GROBID network loop, fresh connection for the write (no idle-in-transaction across the loop over the flaky tunnel).
#5/#10 tex: flatten_inputs strips unresolved input/include at the depth cap (no literal leak on cycles); _norm_texkey strips only a single leading ./ . #6/#7 arxiv.fetch_source: keep non-.tex members resolvable for input; pick primary on an UN-commented documentclass line. #13 is_arxiv_id: also exclude http:// and arXiv-DOI forms. Tests added/updated for each.
Left as deliberate decisions: #3 (pre-section text drop is pre-existing in extract_sections; abstract stored separately), #8 (script normalizer is intentionally self-contained, already documented), #9/#11/#12 (no current trigger / tightening the gzip heuristic would reject valid old LaTeX like documentstyle / title truncation is by design on a write-only column). ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolved 4 conflicts: cli.py (R-D citing-coverage warning + main's small-corpus JSON/hub-title changes both kept); ingest.py (C-7 paper.id=caller-id pin first, then DQ-2 abstract recovery); test_ingest.py + test_openalex.py (kept both branches' tests).
C-7/DQ-5 overlap (both canonicalise papers.id): kept both layers — openalex._canonical_id normalizes fetch_paper's id (DQ-5); ingest pins papers.id to the caller's bare id (C-7); they converge on the bare canonical id. Fixed the auto-merged test_fetch_paper_success that had contradictory bare/URL assertions. 402 tests pass; ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Polish _clean_title for R-F: strip accent/escape macros (backslash-quote o to o, backslash-amp), convert tilde ties to spaces, and truncate at a word boundary so stored section titles read cleanly (mobius invariance, colin de verdiere). Tests added; ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add quality.section_label(title, content): for section-aware .tex ingest, keep the controlled bucket (intro/theorem/proof/abstract/bibliography) when the real \section heading maps to one, else store the cleaned title verbatim (e.g. 'preliminaries', 'rigidity') instead of collapsing to 'body'. Math papers title most sections descriptively, so R-C's vocab-only mapping left ~90% as 'body'; this lifts the section signal toward near-full. .pdf/.txt/run_quality_pass keep content-based classify_section unchanged. Safe because the section column is write-only (no consumer filters on the vocab).
New _clean_title (strip LaTeX/numbering, lower-case, truncate). Tests: section_label bucket/verbatim/fallback paths + _clean_title; .tex ingest stores a descriptive heading as its title. Full suite 377 passed; ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add graph.citing_coverage() (papers with >=1 out-edge / total) and a graph_min_citing_coverage setting (default 0.8). codex graph report now surfaces 'Citing coverage: N/M papers with out-edges (P%)' in text and JSON, and warns when the share is below threshold -- low citing coverage starves PageRank/coupling even when the paper count looks healthy (the DQ-1 sub-item). Separate from the existing total-count graph_min_corpus_size warning.
Tests: citing_coverage unit tests + CLI tests (line shown, warning fires/suppressed, JSON field). Full suite green; ruff + mypy clean. Live: graph report shows 29/29 (100%), no warning.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OpenAlex returns arXiv works under the arXiv DOI (10.48550/arXiv.<id>); _normalize_doi (DQ-5) left it as 10.48550/arxiv.<id>, but the corpus keys arXiv papers on the BARE id (2305.10988 / math/0603097). So re-ingesting an arXiv paper by its bare id missed INSERT ... ON CONFLICT (id) and tripped papers_openalex_id_key — the arXiv half of DQ-5, explicitly deferred there and surfaced again driving the R-C .tex re-ingest.
Add _canonical_id() (normalize DOI, then strip the 10.48550/arxiv. prefix) and use it in _map_paper. Regression tests (modern + legacy arXiv DOI, regular DOI unchanged, fetch_paper bare id). Confirmed live: re-ingesting 2305.10988 now UPDATEs in place. Full suite 370 passed; ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Old arXiv submissions (e.g. legacy math/* papers) are served as a bare gzipped .tex rather than a .tar.gz, which tarfile.open rejected with 'invalid header'. Fall back to gunzipping the body directly when the tar parse fails, so R-C covers those papers too (recovers math/0001176 + math/0306167). Adds a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flatten multi-file arXiv LaTeX (\input/\include) in arxiv.fetch_source so the full body is assembled rather than just the primary file's include skeleton, and add section-aware chunking (tex.chunk_sections) so .tex chunks never span a \section boundary and are labelled by their real heading. The ingest .tex path now produces (section, chunk) pairs, quality-gated together so labels stay aligned; the stored 'section' column gains genuine signal instead of mostly 'body' (serves DQ-3 fidelity + audit R-12).
Adds scripts/rc_tex_reingest.py to re-ingest arXiv papers from .tex (dry-run by default; replaces chunks). Tests: flatten_inputs, chunk_sections, multi-file fetch_source, section-label ingest, is_arxiv_id. Full suite 365 passed; ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add crossref.fetch_references(doi), which parses message.reference[].DOI into Citations (DOI-less book/unstructured refs are skipped — they cannot join the graph). Wire it as the third leg of the ingest citation fallback chain: OpenAlex-empty -> S2 -> Crossref, for DOI papers only (Crossref's works endpoint is DOI-keyed). Cited DOIs are normalized to the canonical bare lower-case form via _norm_cited_id.
As a fallback it fires only when OpenAlex and S2 both return no references, so it adds a forward-looking third source without changing the already-covered corpus.
Tests: fetch_references unit tests (DOI edges / unstructured skipped / no refs / 404) and ingest tests for both fallback directions. Full suite 361 passed; ruff + mypy clean. Live-validated: 33/40/24 refs for Springer/ACM/Wiley DOIs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable roadmap R-A (GROBID reference extraction) end-to-end and run it
against local Jetson infra instead of a Mac/Rosetta emulation.
Backfill:
- Add scripts/ra_grobid_backfill.py: references-only, idempotent citation
backfill (dry-run default). Deliberately not ingest_paper(source_path=pdf),
which re-OCRs and replaces the DQ-3-clean .txt chunks; this touches only the
citations table (ON CONFLICT DO NOTHING). 7 tests.
- Make extract_references timeout configurable (large theses exceed the 60s
default, especially against a slower GROBID).
Jetson infra:
- Bump grobid/grobid 0.8.2 -> 0.9.0-crf. The -crf tag is the only GROBID
variant published as an arm64 multi-arch manifest; every 0.8.x tag and the
full deep-learning image are amd64-only, which is why GROBID never ran on the
aarch64 Jetson. Enable the 4g memory cap + init/ulimits per GROBID docs.
- Add docs/infra/grobid-jetson.md runbook: arm64 image rationale, the two host
prerequisites (docker-group membership + the Compose v2 CLI plugin, both
missing on the Jetson), service-scoped deploy, and the GET /api/isalive check.
Verified live 2026-06-17: native arm64 pull, isalive=true, end-to-end
extract_references on lutz-2024-thesis.pdf = 116 refs (101 with DOI/arXiv).
docs(audit): mark R-A core done (citing coverage 27/29 -> 29/29; edges 920 -> 1022).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GROBID extracts a cited DOI verbatim from PDF reference text, so it can be mixed-case, a https://doi.org/ URL, or doi:-prefixed. The PDF citation branch inserted it unchanged, while every other path (S2 supplement, OpenAlex, papers.id itself) uses the bare lower-cased DOI. Once R-A runs GROBID reference extraction over arXiv PDFs, those un-normalized cited-ids would never equal a bare-lowercase papers.id/cited_id — dangling the cross-citations and splitting one reference into case-/URL-variant graph nodes.
Extend _norm_cited_id to strip https://doi.org//http://doi.org//doi: prefixes and lower-case (mirroring openalex._normalize_doi), making it the single canonical cited-id normalizer, and route the GROBID branch through it. arXiv ids and S2 paperIds still pass through untouched.
Add test_ingest_pdf_grobid_citations_normalize_doi covering mixed-case, URL-form, and doi:-prefixed DOIs plus an arXiv-only ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
openalex._map_paper set Paper.id from the raw `doi` field, which OpenAlex
returns as a full URL (https://doi.org/10.x). The canonical papers.id (M-1
migration) is the bare, lower-cased DOI, so re-ingesting a DOI paper missed
ON CONFLICT (id) and tripped the separate papers_openalex_id_key constraint
(UniqueViolation).
Fix: _normalize_doi() strips https://doi.org//http://doi.org//doi: and
lower-cases; applied in the DOI branch of _map_paper only (W-id fallback
untouched). Also un-breaks _s2_id_for/Crossref recovery, which the URL form
silently defeated. Corrected the _SAMPLE_WORK fixture to the URL form (the bare
fixture hid the bug) + added _normalize_doi and re-ingest idempotency regression
tests.
Live verification: re-ingest of 10.1007/s00454-019-00132-8 (previously crashed)
now UPDATEs in place — single row, added_at unchanged, no stray row, citations
45->45. Unblocks roadmap R-A/R-C. Full suite 348 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:
- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
(authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
Crossref *before* embedding, so the paper gets a real (non-zero) vector.
Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.
Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.
Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses
without a parsed reference list (verified referenced_works_count=0 server-side
for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit.
- ingest.py: when OpenAlex returns no references, fall back to a Semantic
Scholar reference supplement (_s2_reference_supplement); citing_id rewritten
to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant
discarded S2 probe in the OpenAlex-404 arXiv branch.
- semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs
responses (.get("data", []) returns None when the key is present-but-null).
- tests: regression test for the OpenAlex-empty -> S2 path.
- Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge
papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce
theses remain (no references in any source).
- docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a
free-source acquisition roadmap (R-A..R-E).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`codex search paper` crashed on the live DB with "operator does not exist:
vector <-> double precision[]". The pgvector `<->` operator needs its RHS
typed as vector; peer call sites (mcp_server.py, wiki.py) cast but the CLI
did not. Mocked unit tests never exercise real pgvector, so it went unseen.
Add the cast to both `<->` occurrences and a regression guard asserting the
SQL contains `::vector`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review: the two latent upsert conflicts should be observable even where not
auto-fixed.
- bibkey collision (auto-suffixed) now logs a warning so a paper landing as
'BobenkoLutz2023a' instead of '…2023' isn't silent.
- openalex_id collision (two papers claiming the same OpenAlex work — a real data
conflict, not auto-renamable) now raises a clear ValueError naming the
openalex_id and the offending paper, instead of the raw psycopg UniqueViolation.
Regression test added for the openalex_id error path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The id-keyed upsert (ON CONFLICT (id)) does not catch the papers.bibkey UNIQUE
constraint, so a second paper whose auto-generated key matches an existing one
(two same-author/year works → e.g. 'BobenkoLutz2023') failed the whole ingest
with a UniqueViolation. Surfaced by the canonical-id re-ingest: 2305.10988
collided with 2310.17529.
ingest now catches a bibkey UniqueViolation, rolls back, and retries the upsert
with an a/b/c… suffix. Non-bibkey violations re-raise unchanged. Mocks don't
raise, so the existing happy-path tests are untouched; a new test drives the
collision-and-retry path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- C-14: embed.py header now states that only the dense path is wired (search
'hybrid' is dense + Postgres FTS); encode_sparse/encode are reserved for a
future sparse-retrieval layer, not yet consumed.
- U-2: --cite-boost help clarifies it re-ranks within the top results (a
tie-breaker), not a hard re-ranking — matching the small alpha effect.
Accepted (documented heuristics, no change): R-3 conflict detection is a
keyword-only signal labelled as such; R-12 classify_section is mostly 'body'
because chunks are word-windows that rarely start at a section header.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- C-6: the content-5-gram check used 'gram in src' on space-joined strings, so the
first/last gram token could match a prefix/suffix of a longer chunk word
('set' inside 'subset'). Both gram and chunk are now space-padded, so a match
requires whole-token alignment. Regression test added.
- R-2: the reference-list filter's author pattern ('Surname, I.') also matched
'Theorem A.'/'Lemma B.', so theorem-dense chunks could be dropped as a
bibliography. The pattern now excludes common math-structure words.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- U-1: 'graph report' shows the paper title next to each in-KB hub (dangling
OpenAlex-id hubs are flagged), instead of bare ids.
- U-3: '--json' now includes a 'small_corpus_warning' field (null unless below
graph_min_corpus_size) instead of silently dropping the warning.
- C-9: new 'codex discover recommend <id>' wires semanticscholar.fetch_recommendations
(was implemented + tested but unreachable).
- U-4: new 'codex graph cocited <id>' wires graph.find_co_cited (likewise).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- R-4: arxiv.fetch_source had a try/except httpx.RequestError that only re-raised
— removed (no behaviour change, less noise).
- R-5: semanticscholar URLs now quote(paper_id, safe=':') so a legacy-arXiv '/'
(arXiv:math/0603097) doesn't corrupt the path and silently return no references.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
database_url, migration_database_url, mcp_auth_token and the mathpix app
id/key are now pydantic SecretStr, so they render as '**********' in any
repr/log/traceback instead of leaking the plaintext credential. Consumers
(db.get_conn, cli.migrate, mcp._require_http_token, mathpix.extract_formulas)
call .get_secret_value() at the point of use. Tests updated to the SecretStr
type.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The grounding guard tokenized with .lower().split() and split sentences on bare
.!?, so faithful claims were wrongly marked ungrounded: 'volume,' != 'volume',
and a decimal like 'V = 1.5' split into a 1-word fragment '5' that fell below
the 5-content-word floor.
- _content_words now strips edge sentence-punctuation (.,;:!?"') from tokens
while preserving math delimiters ()=+; claim and chunk are normalised
identically so matching stays consistent.
- _SENTENCE_SPLIT_RE splits on .!? only when followed by whitespace/end, so
decimals no longer create spurious sentence boundaries.
Regression test: a 5-word claim with a trailing comma now grounds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- C-3 (MED): wiki_read returned 'sources' = [concept_slug] (a placeholder), not
the cited sources. Now parses the page's inline [BibKey #loc] citations and
returns the actual set of cited bibkeys.
- S-2 (LOW): _TokenAuth compared the Bearer header with != (length/equality
timing side-channel). Use secrets.compare_digest for constant-time comparison.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
find_gaps defaulted its probe topics to paper titles, so a title — which
retrieves mostly its own single bibkey — fell below synthesis_gap_min_coverage
and flagged almost every paper as a 'gap'. Coverage-map gaps now run only for
explicitly-requested topics; the CLI gains a repeatable --topic option. The
code-vs-corpus gap path (precise) is unchanged and still default.
Tests: the two cases that relied on default-title coverage gaps now pass explicit
topics; added a regression that no coverage gaps appear without --topic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- R-9 (MED): _inject_cross_refs now protects LaTeX math spans ($…$, $$…$$,
\(…\), \[…\]) like inline code, so a concept name inside a formula is no longer
rewritten to a [[slug]] link and corrupting the LaTeX. Regression test added.
- C-13 (LOW): mark ungrounded claims on the raw body BEFORE injecting cross-refs
(render then inject), so a concept name inside a claim cannot stop the ⚠ regex
from matching.
- C-12 (LOW): _try_embed_formulas is now a true no-op — it previously issued a
per-concept information_schema round-trip that did nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
M-1: schema.sql could not be re-applied (leading CREATE TABLE/INDEX lacked
IF NOT EXISTS), apply_schema was never called, and the live migration just
failed on a missing chunks.section column. Fixes:
- schema.sql: all CREATE TABLE/INDEX now use IF NOT EXISTS — the whole file is
re-applyable as a no-op.
- codex migrate: new CLI command that applies schema.sql. Connects via
MIGRATION_DATABASE_URL (falls back to DATABASE_URL) and catches
InsufficientPrivilege with guidance — because the app role is DML-only and
core tables are owned by 'postgres' (the privilege dimension found during the
live migration incident).
- config: migration_database_url (optional privileged connection).
- db: apply_schema docstring corrected (idempotency now true) + privilege note.
- T-2: static test that schema.sql is fully idempotent; migrate privilege-error
test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_formulas/extract_figures derive paper_id from Path(pdf_path).stem, which
need not equal papers.id (and won't, given canonical ids). Inserting that stem
violated the formulas/figures -> papers(id) FK on rich (.pdf) ingest. The DELETE
already used paper.id; the INSERTs now do too.
Strengthened the rich-ingest test: the fake formula/figure carry a filename-stem
paper_id distinct from papers.id, and the test asserts the inserted rows are
keyed on papers.id.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
openalex.fetch_paper fills paper.id from OpenAlex's URL-form doi/id
(e.g. https://doi.org/10.48550/arxiv.math/0603097), and ingest upserted that as
papers.id. Live DB confirmed 35/36 ids were URL-form — breaking bare-id lookups
(graph related / discover citing / search by id) and leaving cited_id matching
to rely solely on openalex_id.
ingest now sets paper.id to the caller-supplied id (the arXiv id / DOI the CLI
and ingest scripts use); OpenAlex's work id is retained as openalex_id and in
paper_identifiers. Regression test: fetch_paper returns a URL-form id, papers.id
upsert uses the bare caller id.
T-3: the OpenAlex test fixture used a bare 'doi' (not the URL form the API
emits) and never asserted paper.id, which masked this. Fixture now uses the URL
form and the test pins fetch_paper's mapping.
NOTE: existing rows keep their URL-form ids until re-ingested — run ingest_all.sh
to migrate the 36-paper corpus (chosen migration path).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- D-2 config.py: graph_cite_boost_alpha described the OLD inverted formula
(score = dense * (1 + alpha*pr)); the code divides distance. Document the
actual boosted_distance = distance / (1 + alpha*pr).
- D-4 synthesis.py: write_leads claimed an 'append-only' INDEX.md that keeps
previously-recorded ids; _update_index actually rebuilds from on-disk glob.
Docstring now matches the implementation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
discover.py compared the raw OpenAlex cited_id against papers.id (a DOI/arXiv
id), so an ingested paper cited by its OpenAlex id was mis-reported as a
discovery lead. graph.py already resolved this via papers.openalex_id, but the
fix was never propagated — two 'dangling' implementations disagreed.
Extract the resolution into one shared constant RESOLVED_CITATIONS_SQL in
graph.py; build_citation_graph and all four discover.py queries
(discovery_leads / citing_papers / cited_by / cocited_papers) now go through it,
so the 'what counts as ingested' rule cannot drift again.
Live-DB validated: before, 13 ingested papers leaked into discovery_leads;
after, the real discovery_leads() returns 0 ingested papers among its leads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each stage (find_connections/gaps/improvements/propose_conjectures) numbered its
leads from seq=1, so connection L-0001, gap L-0001 and improvement L-0001 all
resolved to grounded/L-0001.md. The CLI concatenates them
(connections + gaps + improvements) and write_leads writes in order, so later
kinds silently overwrote earlier ones — survivors = max(per-kind count), not the
sum. Reproduced: 4 distinct leads -> 2 files.
_make_lead_id now takes the kind and emits L-C-/L-G-/L-I-/L-X- prefixes, keeping
same-seq ids distinct across stages. Regression tests cover the format and the
no-collision invariant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
compile_all's quarantine gate was guarded by `total_claims > 0`, so a page
with no parseable citations — including the LLM-outage case where
compile_concept returns an empty page — fell through to the publish branch and
was written to wiki/ + marked compiled. Such a page carries zero grounding
evidence.
Now: total_claims == 0 (or grounding_rate below threshold) quarantines and does
NOT update the compile-state, so a transient LLM failure retries next run
instead of freezing an empty/uncited page as 'compiled'. Empty bodies are
recorded but not written as draft files.
Regression test reproduces the exact bypass (uncited LLM output -> quarantined,
not published).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Committed code predated the line-length=100 ruff config; this brings the
five drifted files into compliance. No logic change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes preprint/journal duplicate problem: arXiv papers were ingested
under their preprint OpenAlex ID, but corpus citations reference the
published-version OpenAlex ID → those citations were invisible in the
graph (appeared as dangling).
Changes:
- infra/schema.sql: CREATE TABLE paper_identifiers (paper_id, openalex_id)
with UNIQUE index; seeded from papers.openalex_id on migration
- codex/graph.py: build_citation_graph LEFT JOINs paper_identifiers as
a second resolution path: COALESCE(p.id, pi.paper_id, c.cited_id)
- codex/ingest.py: every ingest inserts openalex_id into paper_identifiers
(ON CONFLICT DO NOTHING) — aliases can be added manually alongside
Live DB: 35 rows seeded + 4 published-version aliases added:
math/0503219 → W2163787581 (DCG 2007)
math/0306167 → W2136126748 (Commun Analysis Geom 2004)
math/0203250 → W1567166970 (Trans AMS 2003)
1005.2698 → W1511400044 (Geom & Topol 2015)
Effect: dangling 574→570; Bobenko+Springborn 2007 now IN-KB rank 9.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build_citation_graph now does:
SELECT c.citing_id, COALESCE(p.id, c.cited_id) AS cited_id
FROM citations c
LEFT JOIN papers p ON p.openalex_id = c.cited_id
Before: cited_id (OpenAlex IDs) never matched papers.id (DOIs) →
13 cross-citations between ingested papers were invisible in graph.
After: in-KB papers use their canonical DOI key; dangling references
keep their raw OpenAlex ID unchanged.
Test: test_cross_citation_uses_doi_when_in_kb (42 passed).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
D-04: add _make_bibkey(paper) — ConcatSurnames+Year, ≤3 authors full list,
>3 authors FirstEtAlYear. Applied in ingest_paper when paper.bibkey is
None. ON CONFLICT now fills NULL bibkeys via COALESCE(papers.bibkey,
EXCLUDED.bibkey) while preserving manually-set values.
D-05b: add codex/__main__.py so `python -m codex` dispatches to the CLI.
D-05c: fix test_ingest.py:75 — rename var to raw_api_citations (the raw
OpenAlex payload with citing_id=openalex_id) and add regression
assertion: citing_id written to DB must equal paper.id (DOI), not
paper.openalex_id.
D-05a (proxy): add TestEntryPoint.test_python_m_codex_help — runs
`sys.executable -m codex --help` via subprocess, covering the
non-uv-run entry path. 253 tests green, ruff+mypy clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs exposed during first batch ingest run after D-03 fix:
1. `.txt` files were silently skipped ("Unbekannter Dateityp") — added
plain-text read path alongside .tex and .pdf.
2. After D-03, OpenAlex resolves arXiv IDs → paper.id uses the DOI
(https://doi.org/…) but fetch_citations emits citing_id=openalex_id
(https://openalex.org/W…). FK constraint citations_citing_id_fkey
→ papers.id violated. Fix: rewrite citing_id to paper.id after
fetching from OpenAlex.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FastMCP defaults to DNS rebinding protection (allowed_hosts=localhost only),
which rejects requests from the Mac (192.168.178.82) to the Jetson
(192.168.178.103:8765). Security is already provided by Bearer token auth
in _TokenAuth middleware and UFW restricting port 8765 to trusted LAN IPs,
making host-header validation redundant in this deployment context.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
F-13 settings (leads_dir, synthesis_llm_*, synthesis_top_k,
synthesis_min_grounded_ratio, synthesis_gap_min_coverage) appeared twice
due to independent commits on F-13 and F-14 branches. Kept the second
(more complete) block; removed the first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lead.__post_init__: raises ValueError on empty provenance list
- find_gaps: no-chunk path now carries topic-marker Provenance instead of []
- _update_index: docstring corrected (rebuilds, not append-only)
- test_model: test_lead_empty_provenance_raises covers the new invariant
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace 3-gram substring match with:
- _last_sentence(): only the last sentence before a citation is checked
(prevents a hallucinated paragraph grounding via a phrase at its end)
- _content_words(): stopwords removed from both claim and chunk before
n-gram comparison (prevents bypass via "the discrete conformal map")
- Content-5-gram: require 5 consecutive non-stopword tokens from last
sentence to appear in the chunk's content-word stream
- Claims with <5 content words in last sentence → ungrounded by default
Add adversarial tests: stopword-3-gram bypass → grounded=False;
legitimate content-5-gram → grounded=True.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
