12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses
without a parsed reference list (verified referenced_works_count=0 server-side
for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit.
- ingest.py: when OpenAlex returns no references, fall back to a Semantic
Scholar reference supplement (_s2_reference_supplement); citing_id rewritten
to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant
discarded S2 probe in the OpenAlex-404 arXiv branch.
- semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs
responses (.get("data", []) returns None when the key is present-but-null).
- tests: regression test for the OpenAlex-empty -> S2 path.
- Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge
papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce
theses remain (no references in any source).
- docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a
free-source acquisition roadmap (R-A..R-E).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`codex search paper` crashed on the live DB with "operator does not exist:
vector <-> double precision[]". The pgvector `<->` operator needs its RHS
typed as vector; peer call sites (mcp_server.py, wiki.py) cast but the CLI
did not. Mocked unit tests never exercise real pgvector, so it went unseen.
Add the cast to both `<->` occurrences and a regression guard asserting the
SQL contains `::vector`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Committed code predated the line-length=100 ruff config; this brings the
five drifted files into compliance. No logic change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_citation_graph now does:
SELECT c.citing_id, COALESCE(p.id, c.cited_id) AS cited_id
FROM citations c
LEFT JOIN papers p ON p.openalex_id = c.cited_id
Before: cited_id (OpenAlex IDs) never matched papers.id (DOIs) →
13 cross-citations between ingested papers were invisible in graph.
After: in-KB papers use their canonical DOI key; dangling references
keep their raw OpenAlex ID unchanged.
Test: test_cross_citation_uses_doi_when_in_kb (42 passed).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
D-04: add _make_bibkey(paper) — ConcatSurnames+Year, ≤3 authors full list,
>3 authors FirstEtAlYear. Applied in ingest_paper when paper.bibkey is
None. ON CONFLICT now fills NULL bibkeys via COALESCE(papers.bibkey,
EXCLUDED.bibkey) while preserving manually-set values.
D-05b: add codex/__main__.py so `python -m codex` dispatches to the CLI.
D-05c: fix test_ingest.py:75 — rename var to raw_api_citations (the raw
OpenAlex payload with citing_id=openalex_id) and add regression
assertion: citing_id written to DB must equal paper.id (DOI), not
paper.openalex_id.
D-05a (proxy): add TestEntryPoint.test_python_m_codex_help — runs
`sys.executable -m codex --help` via subprocess, covering the
non-uv-run entry path. 253 tests green, ruff+mypy clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lead.__post_init__: raises ValueError on empty provenance list
- find_gaps: no-chunk path now carries topic-marker Provenance instead of []
- _update_index: docstring corrected (rebuilds, not append-only)
- test_model: test_lead_empty_provenance_raises covers the new invariant
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace 3-gram substring match with:
- _last_sentence(): only the last sentence before a citation is checked
(prevents a hallucinated paragraph grounding via a phrase at its end)
- _content_words(): stopwords removed from both claim and chunk before
n-gram comparison (prevents bypass via "the discrete conformal map")
- Content-5-gram: require 5 consecutive non-stopword tokens from last
sentence to appear in the chunk's content-word stream
- Claims with <5 content words in last sentence → ungrounded by default
Add adversarial tests: stopword-3-gram bypass → grounded=False;
legitimate content-5-gram → grounded=True.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add _detect_conflicts() using adversative keyword heuristic (but,
however, in contrast, …) between chunks of different bibkeys; results
populate CompileReport.conflicts and appear in wiki/log.md.
Also add CompileReport.quarantined field (used by Fix 2).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_inject_cross_refs temporarily replaces backtick spans with null-byte
placeholders before injecting [[slug]] cross-refs, then restores them.
This prevents concept titles inside `code` from being rewritten.
Add tests for inline-code protection and full-list cross-ref injection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
compile_concept now accepts chunks as a positional parameter; compile_all
retrieves chunks once and passes them directly, avoiding double-retrieve.
Also separates all_concepts (full list) from compile_concepts (filtered),
so cross-ref injection always uses the complete concept list regardless of
--concept filter (Fix 6 + Fix 8).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_parse_claims now skips matches where bibkey contains spaces or starts
with "http", preventing Markdown hyperlinks like [text](https://...) from
being misidentified as citations. Add two tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Implement all CLI commands as specified in F-07: ingest, search, and discover subcommands (leads, citing, cited-by, cocited), and provenance subcommands (scan, add-link, links, bib).
- Add ask command stub that exits with code 1 (not yet implemented).
- Implement comprehensive tests with mocked dependencies (no real DB/API calls).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Review-Gate finding: GROBID emits type="arXiv" (camel-case), not "arxiv".
XPath literal match silently returned empty strings for all real responses.
Fix: iterate idno elements and compare .lower() == "arxiv".
Test fixture updated to reflect real GROBID output (type="arXiv").
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
