Commit Graph

19 Commits

Author SHA1 Message Date
Tarik Moussa
9e5184385a merge: reconcile data-quality roadmap with audit-remediation main (#12-15)
Resolved 4 conflicts: cli.py (R-D citing-coverage warning + main's small-corpus JSON/hub-title changes both kept); ingest.py (C-7 paper.id=caller-id pin first, then DQ-2 abstract recovery); test_ingest.py + test_openalex.py (kept both branches' tests).

C-7/DQ-5 overlap (both canonicalise papers.id): kept both layers — openalex._canonical_id normalizes fetch_paper's id (DQ-5); ingest pins papers.id to the caller's bare id (C-7); they converge on the bare canonical id. Fixed the auto-merged test_fetch_paper_success that had contradictory bare/URL assertions. 402 tests pass; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 15:11:26 +02:00
Tarik Moussa
5672358bbe feat(graph): warn on low citing-paper coverage in graph report (R-D)
Add graph.citing_coverage() (papers with >=1 out-edge / total) and a graph_min_citing_coverage setting (default 0.8). codex graph report now surfaces 'Citing coverage: N/M papers with out-edges (P%)' in text and JSON, and warns when the share is below threshold -- low citing coverage starves PageRank/coupling even when the paper count looks healthy (the DQ-1 sub-item). Separate from the existing total-count graph_min_corpus_size warning.

Tests: citing_coverage unit tests + CLI tests (line shown, warning fires/suppressed, JSON field). Full suite green; ruff + mypy clean. Live: graph report shows 29/29 (100%), no warning.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 07:59:25 +02:00
Tarik Moussa
553ae3995d fix(search): cast query embedding to ::vector in search paper (DQ-4)
`codex search paper` crashed on the live DB with "operator does not exist:
vector <-> double precision[]". The pgvector `<->` operator needs its RHS
typed as vector; peer call sites (mcp_server.py, wiki.py) cast but the CLI
did not. Mocked unit tests never exercise real pgvector, so it went unseen.

Add the cast to both `<->` occurrences and a regression guard asserting the
SQL contains `::vector`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 23:51:38 +02:00
Tarik Moussa
45c3b16dd7 docs: clarify sparse-not-wired and cite-boost tie-breaker (audit C-14, U-2)
- C-14: embed.py header now states that only the dense path is wired (search
  'hybrid' is dense + Postgres FTS); encode_sparse/encode are reserved for a
  future sparse-retrieval layer, not yet consumed.
- U-2: --cite-boost help clarifies it re-ranks within the top results (a
  tie-breaker), not a hard re-ranking — matching the small alpha effect.

Accepted (documented heuristics, no change): R-3 conflict detection is a
keyword-only signal labelled as such; R-12 classify_section is mostly 'body'
because chunks are word-windows that rarely start at a section header.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 21:47:40 +02:00
Tarik Moussa
92928bab99 feat(cli): graph-report titles + JSON warning; wire recommend/cocited (U-1,U-3,U-4,C-9)
- U-1: 'graph report' shows the paper title next to each in-KB hub (dangling
  OpenAlex-id hubs are flagged), instead of bare ids.
- U-3: '--json' now includes a 'small_corpus_warning' field (null unless below
  graph_min_corpus_size) instead of silently dropping the warning.
- C-9: new 'codex discover recommend <id>' wires semanticscholar.fetch_recommendations
  (was implemented + tested but unreachable).
- U-4: new 'codex graph cocited <id>' wires graph.find_co_cited (likewise).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 21:43:19 +02:00
Tarik Moussa
b4fbd7930e feat(config): wrap credentials in SecretStr (audit S-4)
database_url, migration_database_url, mcp_auth_token and the mathpix app
id/key are now pydantic SecretStr, so they render as '**********' in any
repr/log/traceback instead of leaking the plaintext credential. Consumers
(db.get_conn, cli.migrate, mcp._require_http_token, mathpix.extract_formulas)
call .get_secret_value() at the point of use. Tests updated to the SecretStr
type.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 21:32:13 +02:00
Tarik Moussa
c4106b0f51 fix(synthesis): make coverage-map gaps opt-in (audit R-8)
find_gaps defaulted its probe topics to paper titles, so a title — which
retrieves mostly its own single bibkey — fell below synthesis_gap_min_coverage
and flagged almost every paper as a 'gap'. Coverage-map gaps now run only for
explicitly-requested topics; the CLI gains a repeatable --topic option. The
code-vs-corpus gap path (precise) is unchanged and still default.

Tests: the two cases that relied on default-title coverage gaps now pass explicit
topics; added a regression that no coverage gaps appear without --topic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 16:32:01 +02:00
Tarik Moussa
cdb7d254df feat(db): idempotent schema + 'codex migrate' command (audit M-1, T-2)
M-1: schema.sql could not be re-applied (leading CREATE TABLE/INDEX lacked
IF NOT EXISTS), apply_schema was never called, and the live migration just
failed on a missing chunks.section column. Fixes:

- schema.sql: all CREATE TABLE/INDEX now use IF NOT EXISTS — the whole file is
  re-applyable as a no-op.
- codex migrate: new CLI command that applies schema.sql. Connects via
  MIGRATION_DATABASE_URL (falls back to DATABASE_URL) and catches
  InsufficientPrivilege with guidance — because the app role is DML-only and
  core tables are owned by 'postgres' (the privilege dimension found during the
  live migration incident).
- config: migration_database_url (optional privileged connection).
- db: apply_schema docstring corrected (idempotency now true) + privilege note.
- T-2: static test that schema.sql is fully idempotent; migrate privilege-error
  test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 15:53:25 +02:00
Tarik Moussa
1f1e0e82b3 fix(F-15): address Opus review findings (CRITICAL + WARN)
CRITICAL:
- cli.py: invert cite-boost formula: divide distance by (1 + alpha*pr)
  instead of multiply — high-PageRank papers now correctly rank higher
- cli.py: wire graph_min_corpus_size config into graph report; warning
  emitted to stderr so JSON stdout stays parseable

WARN:
- graph.py: document that damping is ignored in small-graph uniform branch
- cli.py: sort dangling citations before slicing for stable output
- cli.py: replace raise typer.Exit(0) with return in empty-graph path
- tests: fix *extra_args helper signatures; add cite-boost ordering test
  and small-corpus warning test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 03:40:30 +02:00
Tarik Moussa
fd51b70000 feat(F-15): citation graph — PageRank, coupling, co-citation, CLI
- codex/graph.py: build_citation_graph (DiGraph from DB), citation_pagerank
  (graceful uniform fallback < 5 nodes), find_related (bibliographic coupling),
  find_co_cited, dangling_citations
- codex/config.py: graph_cite_boost_alpha=0.3, graph_min_corpus_size=15
- codex/cli.py: graph report [--top-n] [--json], graph related <paper-id>
  [--min-shared]; search paper --cite-boost (PageRank score weighting)
- tests/graph/: 39 tests (graph functions + CLI) — 326 total green

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 03:31:36 +02:00
9647897173 feat(F-16): chunk quality gate + section classification
3-signal quality filter (length/alpha-ratio/bib-score) + rule-based section classifier + retroactive CLI pass. 35 new tests, 287 total green.
2026-06-15 01:22:44 +00:00
Tarik Moussa
8cf0cc7e01 feat(synthesis): Lead/Provenance model + grounded leads + conjecture generator
- codex/synthesis.py: Lead/Provenance dataclasses, find_connections/gaps/improvements,
  propose_conjectures (status=unverified, quarantined in leads/conjectures/),
  write_leads (grounded→leads/grounded/, conjectures→leads/conjectures/ HARD invariant)
- codex/cli.py: synthesis leads/conjectures/report command group
- codex/config.py: F-13 settings (leads_dir, synthesis_llm_*, synthesis_top_k,
  synthesis_min_grounded_ratio, synthesis_gap_min_coverage)
- tests/synthesis/: 42 tests covering model, grounding, conjecture invariant, quarantine, CLI
- spike/: F-13 spike script + output (live run attempted)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-14 08:37:42 +02:00
Tarik Moussa
408e4886bb fix(wiki): basic conflict detection in CompileReport
Add _detect_conflicts() using adversative keyword heuristic (but,
however, in contrast, …) between chunks of different bibkeys; results
populate CompileReport.conflicts and appear in wiki/log.md.
Also add CompileReport.quarantined field (used by Fix 2).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 05:11:06 +02:00
Tarik Moussa
1df9be6563 feat(F-09): rich parsing — formula + figure extraction
- codex/parsing/mathpix.py: pix2tex (local, CPU) primary + MathPix API
  optional; bbox heuristic h>15px, math-char-count>5; singleton model cache
- codex/parsing/figures.py: pymupdf embedded-image extraction → PNG;
  caption detection via proximity + "Figure/Fig./Abbildung" prefix
- codex/models.py: FormulaChunk + FigureChunk dataclasses (R-10/R-11)
- codex/ingest.py: --rich flag wires formula+figure extraction post-ingest
- codex/cli.py: search_app sub-typer (paper + formula subcommands),
  --rich flag on ingest; wiki_app from F-12 preserved intact
- codex/config.py: mathpix_app_id/key, pix2tex_fallback, figures_dir
- infra/schema.sql: formulas + figures tables with HNSW pgvector indexes
- pyproject.toml: pymupdf>=1.24, pix2tex>=0.1.4
- tests/parsing/test_mathpix.py + test_figures.py: 31 tests (mock pix2tex
  + MathPix HTTP, real pymupdf on synthetic PDF)

Gate: 158 passed, ruff clean, mypy clean (20 files)
Requirements: R-10 R-11 R-12 R-13 R-14 → done

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 01:16:24 +02:00
Tarik Moussa
d2f9141a5c fix(cli): remove F-09 spillover from F-12 branch, keep only wiki group
cli.py and config.py were accidentally contaminated with uncommitted F-09
changes (search_app restructure, rich ingest, mathpix/figures settings).
This restores both files to main-baseline + F-12-only additions:
- cli.py: wiki_app group (compile/list/check), restore @app.command search
- config.py: wiki_dir, wiki_llm_model, wiki_llm_url, wiki_top_k only

All 127 tests green, ruff+mypy clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 20:28:22 +02:00
Tarik Moussa
4d361fd8dc feat(cli): wiki compile/list/check command group (F-12)
Adds codex wiki compile/list/check Typer subgroup to cli.py.
Adds wiki_dir, wiki_llm_model, wiki_llm_url, wiki_top_k settings
to config.py (F-12 section only).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 20:22:55 +02:00
Tarik Moussa
2acb2ee213 feat(cli): wire all commands via Typer (ingest, search, discover, provenance, ask-stub)
- Implement all CLI commands as specified in F-07: ingest, search, and discover subcommands (leads, citing, cited-by, cocited), and provenance subcommands (scan, add-link, links, bib).
- Add ask command stub that exits with code 1 (not yet implemented).
- Implement comprehensive tests with mocked dependencies (no real DB/API calls).

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-06-05 07:22:03 +02:00
Tarik Moussa
b52a6a7412 fix: add @app.callback so codex --help exits 0
Bare Typer() with no commands raised RuntimeError at runtime.
Callback makes the entry point callable without exposing F-07 logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 11:26:35 +02:00
Tarik Moussa
1dd7a4bea3 fix: add cli.py stub so entry point resolves before F-07
Review-Gate finding: pyproject.toml [project.scripts] referenced
codex.cli:app which was absent; uv run codex raised ModuleNotFoundError.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 11:25:17 +02:00