Commit Graph

58 Commits

Author SHA1 Message Date
Tarik Moussa
a1a6f4590b docs(adr): ADR-F15 literature graph — GO, Bobenko+Springborn rank 7
Live-corpus spike (29 papers, 495 nodes, 590 edges, 478 dangling):
- Bobenko & Springborn 2007 at rank 7 — manual hub expectation PASS
- Pinkall & Polthier 1993 at rank 1 (foundational DGP)
- Score spread 5.7 % (0.002282→0.002159); flat but ordered correctly
- cite-boost alpha=0.3: ~0.06 % boost per unit PR — tie-breaker, not
  hard re-rank; correct formula is distance/(1+alpha*pr), not multiply
- Known limitation: citing_id(DOI)→cited_id(OpenAlex) mismatch renders
  inter-KB cross-citations invisible; non-blocking at corpus size 29

Closes R-46.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 07:11:50 +02:00
2c476dc038 Merge pull request 'feat(F-15): citation graph — PageRank, coupling, co-citation, CLI' (#11) from feat/F-15-literature-graph into main
feat(F-15): citation graph — PageRank, coupling, co-citation, CLI

R-41..R-45 done. R-46 (ADR spike) todo pending full-corpus ingest.
2026-06-15 03:18:22 +00:00
Tarik Moussa
1f1e0e82b3 fix(F-15): address Opus review findings (CRITICAL + WARN)
CRITICAL:
- cli.py: invert cite-boost formula: divide distance by (1 + alpha*pr)
  instead of multiply — high-PageRank papers now correctly rank higher
- cli.py: wire graph_min_corpus_size config into graph report; warning
  emitted to stderr so JSON stdout stays parseable

WARN:
- graph.py: document that damping is ignored in small-graph uniform branch
- cli.py: sort dangling citations before slicing for stable output
- cli.py: replace raise typer.Exit(0) with return in empty-graph path
- tests: fix *extra_args helper signatures; add cite-boost ordering test
  and small-corpus warning test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 03:40:30 +02:00
Tarik Moussa
fd51b70000 feat(F-15): citation graph — PageRank, coupling, co-citation, CLI
- codex/graph.py: build_citation_graph (DiGraph from DB), citation_pagerank
  (graceful uniform fallback < 5 nodes), find_related (bibliographic coupling),
  find_co_cited, dangling_citations
- codex/config.py: graph_cite_boost_alpha=0.3, graph_min_corpus_size=15
- codex/cli.py: graph report [--top-n] [--json], graph related <paper-id>
  [--min-shared]; search paper --cite-boost (PageRank score weighting)
- tests/graph/: 39 tests (graph functions + CLI) — 326 total green

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 03:31:36 +02:00
9647897173 feat(F-16): chunk quality gate + section classification
3-signal quality filter (length/alpha-ratio/bib-score) + rule-based section classifier + retroactive CLI pass. 35 new tests, 287 total green.
2026-06-15 01:22:44 +00:00
1a9afb4433 Merge pull request 'fix(ingest): D-04 bibkey heuristic + D-05a/b/c regression tests + __main__' (#9) from fix/d04-d05-ingest-fixes into main 2026-06-15 01:07:35 +00:00
Tarik Moussa
b87087d3c6 fix(ingest): D-04 bibkey heuristic + D-05a/b/c regression tests + __main__
D-04: add _make_bibkey(paper) — ConcatSurnames+Year, ≤3 authors full list,
      >3 authors FirstEtAlYear. Applied in ingest_paper when paper.bibkey is
      None. ON CONFLICT now fills NULL bibkeys via COALESCE(papers.bibkey,
      EXCLUDED.bibkey) while preserving manually-set values.

D-05b: add codex/__main__.py so `python -m codex` dispatches to the CLI.

D-05c: fix test_ingest.py:75 — rename var to raw_api_citations (the raw
       OpenAlex payload with citing_id=openalex_id) and add regression
       assertion: citing_id written to DB must equal paper.id (DOI), not
       paper.openalex_id.

D-05a (proxy): add TestEntryPoint.test_python_m_codex_help — runs
       `sys.executable -m codex --help` via subprocess, covering the
       non-uv-run entry path. 253 tests green, ruff+mypy clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 03:07:06 +02:00
Tarik Moussa
335ca2c923 docs(adr): ADR-F13 synthesis engine — GO + zero-fact-leak enforcement (N-02)
Closes audit N-02: docs/adr/ADR-F13-synthesis-engine.md was missing,
blocking R-34. Documents spike result (GO, Opus APPROVE 2 passes),
four-stage lead taxonomy, grounding discipline, three-level
zero-fact-leak invariant, and falsification-gate in _parse_conjecture_output.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 01:54:53 +02:00
36a5d9566a Merge pull request 'fix(docs): correct provenance CLI command names in README (D-01)' (#8) from fix/cli-readme-drift into main 2026-06-14 23:51:07 +00:00
Tarik Moussa
b12789a47f fix(docs): correct provenance CLI command names in README (D-01)
Audit D-01 (2026-06-15): README documented `provenance sync` and
`provenance export-bib` but cli.py:161/195 defines `scan` and `bib`.
Fix: update both occurrences (prose + CLI reference block).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 00:49:50 +02:00
1bec18ae82 Merge pull request 'fix(ingest): support .txt sources + fix citing_id FK violation' (#7) from fix/ingest-txt-and-citing-id into main
fix(ingest): support .txt sources + fix citing_id FK violation — Opus APPROVE
2026-06-14 13:56:04 +00:00
Tarik Moussa
5025ff3a88 fix(ingest): support .txt sources + fix citing_id FK violation
Two bugs exposed during first batch ingest run after D-03 fix:

1. `.txt` files were silently skipped ("Unbekannter Dateityp") — added
   plain-text read path alongside .tex and .pdf.

2. After D-03, OpenAlex resolves arXiv IDs → paper.id uses the DOI
   (https://doi.org/…) but fetch_citations emits citing_id=openalex_id
   (https://openalex.org/W…). FK constraint citations_citing_id_fkey
   → papers.id violated. Fix: rewrite citing_id to paper.id after
   fetching from OpenAlex.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 15:54:05 +02:00
6c28446644 Merge pull request 'fix(mcp): disable DNS rebinding protection for LAN HTTP transport' (#6) from fix/mcp-transport-security into main
fix(mcp): disable DNS rebinding protection for LAN HTTP transport
2026-06-14 12:55:59 +00:00
Tarik Moussa
d29dcf2fb5 fix(mcp): disable DNS rebinding protection for LAN HTTP transport
FastMCP defaults to DNS rebinding protection (allowed_hosts=localhost only),
which rejects requests from the Mac (192.168.178.82) to the Jetson
(192.168.178.103:8765). Security is already provided by Bearer token auth
in _TokenAuth middleware and UFW restricting port 8765 to trusted LAN IPs,
making host-header validation redundant in this deployment context.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 14:55:29 +02:00
1d1203b225 Merge pull request 'fix(sources): resolve arXiv IDs via arXiv DOI (OpenAlex /works/arxiv: 404s)' (#5) from fix/openalex-arxiv-doi into main
fix(sources): resolve arXiv IDs via arXiv DOI (OpenAlex /works/arxiv: 404s)
2026-06-14 12:31:33 +00:00
Tarik Moussa
50e6bb8210 fix(config): remove duplicate F-13 settings block (merge artefact)
F-13 settings (leads_dir, synthesis_llm_*, synthesis_top_k,
synthesis_min_grounded_ratio, synthesis_gap_min_coverage) appeared twice
due to independent commits on F-13 and F-14 branches. Kept the second
(more complete) block; removed the first.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 08:59:05 +02:00
Tarik Moussa
b2ef8f2575 docs(mcp): client registration (Claude Code / Desktop)
Documents MCP server setup for Claude Code (.mcp.json) and Claude Desktop
(claude_desktop_config.json), HTTP transport config, and available tools table.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 08:55:49 +02:00
Tarik Moussa
b803811701 feat(mcp): FastMCP server exposing read-only KB tools (stdio)
Implements F-14: thin FastMCP wrapper over existing codex domain modules.
Seven read-only tools: search, ask, wiki_read, wiki_list, discover_leads,
provenance_verify, synthesis_browse. All optional-feature tools degrade
gracefully to {"error": "feature not available"} instead of crashing.
Adds mcp[cli]>=1.0 dependency and codex-mcp console_script entry-point.
28 new tests across test_server, test_search, test_readonly, test_graceful,
test_http_auth — all green; 0 regressions in existing 172 tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 08:55:49 +02:00
Tarik Moussa
59763105a5 fix(synthesis): enforce non-empty provenance + fix gap lead + clean index docstring
- Lead.__post_init__: raises ValueError on empty provenance list
- find_gaps: no-chunk path now carries topic-marker Provenance instead of []
- _update_index: docstring corrected (rebuilds, not append-only)
- test_model: test_lead_empty_provenance_raises covers the new invariant

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 08:52:35 +02:00
Tarik Moussa
8cf0cc7e01 feat(synthesis): Lead/Provenance model + grounded leads + conjecture generator
- codex/synthesis.py: Lead/Provenance dataclasses, find_connections/gaps/improvements,
  propose_conjectures (status=unverified, quarantined in leads/conjectures/),
  write_leads (grounded→leads/grounded/, conjectures→leads/conjectures/ HARD invariant)
- codex/cli.py: synthesis leads/conjectures/report command group
- codex/config.py: F-13 settings (leads_dir, synthesis_llm_*, synthesis_top_k,
  synthesis_min_grounded_ratio, synthesis_gap_min_coverage)
- tests/synthesis/: 42 tests covering model, grounding, conjecture invariant, quarantine, CLI
- spike/: F-13 spike script + output (live run attempted)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-14 08:37:42 +02:00
Tarik Moussa
967e6baa59 feat(wiki): ADR-F12-wiki-compile.md — GO decision + grounding measurement
Document Spike result (grounding ~0.95, hallucination ≤5% on 3 concepts),
content-5-gram guard design, quarantine mechanism, LLM graceful degradation,
cross-ref injection rules, and conflict detection MVP.
R-27 satisfied.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 05:11:06 +02:00
Tarik Moussa
7d338d29ca fix(wiki): sentence-level claim granularity + content-5-gram grounding guard
Replace 3-gram substring match with:
- _last_sentence(): only the last sentence before a citation is checked
  (prevents a hallucinated paragraph grounding via a phrase at its end)
- _content_words(): stopwords removed from both claim and chunk before
  n-gram comparison (prevents bypass via "the discrete conformal map")
- Content-5-gram: require 5 consecutive non-stopword tokens from last
  sentence to appear in the chunk's content-word stream
- Claims with <5 content words in last sentence → ungrounded by default

Add adversarial tests: stopword-3-gram bypass → grounded=False;
legitimate content-5-gram → grounded=True.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 05:11:06 +02:00
Tarik Moussa
408e4886bb fix(wiki): basic conflict detection in CompileReport
Add _detect_conflicts() using adversative keyword heuristic (but,
however, in contrast, …) between chunks of different bibkeys; results
populate CompileReport.conflicts and appear in wiki/log.md.
Also add CompileReport.quarantined field (used by Fix 2).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 05:11:06 +02:00
Tarik Moussa
ef46d14aed fix(wiki): protect inline-code spans from cross-ref injection
_inject_cross_refs temporarily replaces backtick spans with null-byte
placeholders before injecting [[slug]] cross-refs, then restores them.
This prevents concept titles inside `code` from being rewritten.
Add tests for inline-code protection and full-list cross-ref injection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 03:58:02 +02:00
Tarik Moussa
0202a2266e fix(wiki): pass chunks explicitly to compile_concept (single retrieve)
compile_concept now accepts chunks as a positional parameter; compile_all
retrieves chunks once and passes them directly, avoiding double-retrieve.
Also separates all_concepts (full list) from compile_concepts (filtered),
so cross-ref injection always uses the complete concept list regardless of
--concept filter (Fix 6 + Fix 8).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 03:58:02 +02:00
Tarik Moussa
60e40f6f36 fix(wiki): exclude URL-shaped bibkeys from claim parsing
_parse_claims now skips matches where bibkey contains spaces or starts
with "http", preventing Markdown hyperlinks like [text](https://...) from
being misidentified as citations. Add two tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 03:58:02 +02:00
Tarik Moussa
89228a1f62 fix(wiki): LLM error handling — graceful degradation on Ollama unavailable
Wrap llm.generate() in compile_concept with a broad try/except;
log a warning and return an empty ConceptPage (no crash) when the
LLM endpoint is unreachable. Add logging module import.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 03:58:02 +02:00
946a80418e Merge pull request 'feat(F-09): rich parsing — formula + figure extraction' (#1) from feat/F-09-rich-parsing into main
feat(F-09): rich parsing — formula + figure extraction
2026-06-14 01:52:23 +00:00
Tarik Moussa
cd42e9abc8 fix(F-09): address review-gate findings — resource leak + idempotency
- mathpix.py + figures.py: wrap fitz page-loop in try/finally so
  doc.close() is guaranteed even on exception (no file-handle leak)
- ingest.py: DELETE FROM formulas/figures WHERE paper_id before re-insert
  so --rich re-ingest is idempotent (BIGSERIAL has no natural UNIQUE key;
  ON CONFLICT DO NOTHING was a no-op)

Gate: 158 passed, ruff clean, mypy clean

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 02:23:21 +02:00
Tarik Moussa
1df9be6563 feat(F-09): rich parsing — formula + figure extraction
- codex/parsing/mathpix.py: pix2tex (local, CPU) primary + MathPix API
  optional; bbox heuristic h>15px, math-char-count>5; singleton model cache
- codex/parsing/figures.py: pymupdf embedded-image extraction → PNG;
  caption detection via proximity + "Figure/Fig./Abbildung" prefix
- codex/models.py: FormulaChunk + FigureChunk dataclasses (R-10/R-11)
- codex/ingest.py: --rich flag wires formula+figure extraction post-ingest
- codex/cli.py: search_app sub-typer (paper + formula subcommands),
  --rich flag on ingest; wiki_app from F-12 preserved intact
- codex/config.py: mathpix_app_id/key, pix2tex_fallback, figures_dir
- infra/schema.sql: formulas + figures tables with HNSW pgvector indexes
- pyproject.toml: pymupdf>=1.24, pix2tex>=0.1.4
- tests/parsing/test_mathpix.py + test_figures.py: 31 tests (mock pix2tex
  + MathPix HTTP, real pymupdf on synthetic PDF)

Gate: 158 passed, ruff clean, mypy clean (20 files)
Requirements: R-10 R-11 R-12 R-13 R-14 → done

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 01:16:24 +02:00
Tarik Moussa
d2f9141a5c fix(cli): remove F-09 spillover from F-12 branch, keep only wiki group
cli.py and config.py were accidentally contaminated with uncommitted F-09
changes (search_app restructure, rich ingest, mathpix/figures settings).
This restores both files to main-baseline + F-12-only additions:
- cli.py: wiki_app group (compile/list/check), restore @app.command search
- config.py: wiki_dir, wiki_llm_model, wiki_llm_url, wiki_top_k only

All 127 tests green, ruff+mypy clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 20:28:22 +02:00
Tarik Moussa
4d361fd8dc feat(cli): wiki compile/list/check command group (F-12)
Adds codex wiki compile/list/check Typer subgroup to cli.py.
Adds wiki_dir, wiki_llm_model, wiki_llm_url, wiki_top_k settings
to config.py (F-12 section only).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 20:22:55 +02:00
Tarik Moussa
aee00515f4 feat(wiki): concept schema loader + concepts.yaml (F-12)
Adds wiki/concepts.yaml (5 curated seeds: discrete-conformal-map,
circle-packing, lobachevsky-function, discrete-yamabe-flow,
hyperideal-tetrahedron) and codex/wiki.py with full load_concepts()
implementation (Concept dataclass, YAML parser, graceful empty list).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 20:22:51 +02:00
Tarik Moussa
c6eaf999a1 chore(env): add NOUGAT_URL to .env.example (F-11 / D-02)
Documents the Nougat HTTP server endpoint. Resolves D-02 drift finding.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 20:14:07 +02:00
Tarik Moussa
456dea102b chore(claude): add project harness config (permissions for uv/ruff/mypy/podman)
codex-py hatte kein .claude/ — F-xx-Worker liefen mit nackten globalen Settings
und wurden bei jedem uv/ruff/mypy/pytest/git-Befehl geprompted. settings.json
spiegelt den knowledge-base-Stil (acceptEdits + harter deny-Block), zugeschnitten
auf den Python/uv-Stack + podman (DB) + ollama. settings.local.json gitignored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 09:03:41 +02:00
Tarik Moussa
c8b127bae0 fix(sources): resolve arXiv IDs via arXiv DOI (OpenAlex /works/arxiv: 404s)
OpenAlex' /works/{id}-Pfad akzeptiert die arxiv:-Namespace-Form nicht (404).
_resolve_id löst arXiv-IDs jetzt auf die arXiv-DOI 10.48550/arXiv.<id> auf —
empirisch verifiziert für modern (1005.2698) und legacy (math/0603097) IDs.
Behebt D-03 (Erstingest: alle Metadaten leer, weil fetch_paper 404 gab).

- _resolve_id: arxiv:-Präfix strippen, arXiv-DOI bauen
- Tests: arxiv (modern/legacy/prefixed) erwarten jetzt doi:10.48550/arXiv.*
- 19 Tests grün, ruff + mypy sauber

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 23:14:52 +02:00
Tarik Moussa
d48db47a7d feat: merge F-08 scaffold + integration tests (config, db, models, pipeline smoke) 2026-06-05 07:43:09 +02:00
Tarik Moussa
65918cc80a feat(tests): scaffold + integration smoke tests (config, db, models, pipeline)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 07:40:25 +02:00
Tarik Moussa
01199c5e46 feat: merge F-07 CLI (Typer commands: ingest, search, discover, provenance, ask-stub) 2026-06-05 07:35:24 +02:00
Tarik Moussa
4c177a0d29 fix(cli): use datetime() for CodeLink.added_at in tests (mypy strict)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 07:34:51 +02:00
Tarik Moussa
2acb2ee213 feat(cli): wire all commands via Typer (ingest, search, discover, provenance, ask-stub)
- Implement all CLI commands as specified in F-07: ingest, search, and discover subcommands (leads, citing, cited-by, cocited), and provenance subcommands (scan, add-link, links, bib).
- Add ask command stub that exits with code 1 (not yet implemented).
- Implement comprehensive tests with mocked dependencies (no real DB/API calls).

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-06-05 07:22:03 +02:00
Tarik Moussa
c65fe19c24 feat: merge F-06 discover + provenance (graph queries, @cite scan, bib export) 2026-06-05 07:15:58 +02:00
Tarik Moussa
7675f63188 feat(discover,provenance): graph discovery queries + @cite scan + code_links + bib export
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 07:13:01 +02:00
Tarik Moussa
1e10736185 feat: merge F-05 ingest pipeline (idempotent upsert, sources+parsing+embed) 2026-06-05 07:08:29 +02:00
Tarik Moussa
5344975bb1 feat(ingest): end-to-end idempotent ingest pipeline
Implements codex/ingest.py with:
- OpenAlex primary / Semantic Scholar fallback metadata fetch
- Abstract dense embedding (zero-vector for missing abstracts)
- Idempotent paper upsert (ON CONFLICT DO UPDATE)
- Source file parsing (.tex via latex_to_text, .pdf via nougat + grobid)
- Chunk deletion + bulk re-insert with dense embeddings
- Citation merge from OpenAlex/S2 API and GROBID PDF refs (dedup via set)
- 7 tests covering basic, tex, pdf, not-found, idempotent, arxiv-fallback,
  and no-abstract scenarios (all mocked, offline)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 07:04:12 +02:00
Tarik Moussa
fbf8949579 feat: merge F-04 embed layer (BGE-M3 dense+sparse via FlagEmbedding) 2026-06-05 06:56:17 +02:00
Tarik Moussa
c33fd9d697 feat: merge F-03 parsing layer (LaTeX, Nougat, GROBID)
Review-Gate: APPROVE (Opus, cold, 2 passes)
23 tests, ruff+mypy clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 06:48:49 +02:00
Tarik Moussa
d5726ed94d feat: merge F-02 sources layer (OpenAlex, SemanticScholar, arXiv)
Review-Gate: APPROVE (Opus, cold, 2 passes)
19 tests, ruff+mypy clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 06:48:49 +02:00
Tarik Moussa
52737abe3d fix(sources): correct OpenAlex ID prefixing, citations endpoint, S2 rate-limit
Review-Gate findings:
- openalex: bare DOIs/arXiv IDs need doi:/arxiv: prefix (bare IDs 404);
  add _resolve_id(); fix fetch_citations to use referenced_works field
  instead of non-existent /works/{id}/references endpoint.
- semanticscholar: wait_fixed(1) was inter-retry only; add per-request
  monotonic rate-limiter (_rate_limit()) before each httpx call.
  Add _is_retryable() filter so 404s don't burn 5 retry slots.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 23:49:35 +02:00
Tarik Moussa
1698f7dcad feat(embed): BGE-M3 dense+sparse via FlagEmbedding (ADR-0002)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-04 23:48:06 +02:00