Commit Graph

4 Commits

Author SHA1 Message Date
Tarik Moussa
c36e7c6ee7 fix(ingest): canonicalise papers.id to the caller id (audit C-7, T-3)
openalex.fetch_paper fills paper.id from OpenAlex's URL-form doi/id
(e.g. https://doi.org/10.48550/arxiv.math/0603097), and ingest upserted that as
papers.id. Live DB confirmed 35/36 ids were URL-form — breaking bare-id lookups
(graph related / discover citing / search by id) and leaving cited_id matching
to rely solely on openalex_id.

ingest now sets paper.id to the caller-supplied id (the arXiv id / DOI the CLI
and ingest scripts use); OpenAlex's work id is retained as openalex_id and in
paper_identifiers. Regression test: fetch_paper returns a URL-form id, papers.id
upsert uses the bare caller id.

T-3: the OpenAlex test fixture used a bare 'doi' (not the URL form the API
emits) and never asserted paper.id, which masked this. Fixture now uses the URL
form and the test pins fetch_paper's mapping.

NOTE: existing rows keep their URL-form ids until re-ingested — run ingest_all.sh
to migrate the 36-paper corpus (chosen migration path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 12:06:53 +02:00
Tarik Moussa
c8b127bae0 fix(sources): resolve arXiv IDs via arXiv DOI (OpenAlex /works/arxiv: 404s)
OpenAlex' /works/{id}-Pfad akzeptiert die arxiv:-Namespace-Form nicht (404).
_resolve_id löst arXiv-IDs jetzt auf die arXiv-DOI 10.48550/arXiv.<id> auf —
empirisch verifiziert für modern (1005.2698) und legacy (math/0603097) IDs.
Behebt D-03 (Erstingest: alle Metadaten leer, weil fetch_paper 404 gab).

- _resolve_id: arxiv:-Präfix strippen, arXiv-DOI bauen
- Tests: arxiv (modern/legacy/prefixed) erwarten jetzt doi:10.48550/arXiv.*
- 19 Tests grün, ruff + mypy sauber

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 23:14:52 +02:00
Tarik Moussa
52737abe3d fix(sources): correct OpenAlex ID prefixing, citations endpoint, S2 rate-limit
Review-Gate findings:
- openalex: bare DOIs/arXiv IDs need doi:/arxiv: prefix (bare IDs 404);
  add _resolve_id(); fix fetch_citations to use referenced_works field
  instead of non-existent /works/{id}/references endpoint.
- semanticscholar: wait_fixed(1) was inter-retry only; add per-request
  monotonic rate-limiter (_rate_limit()) before each httpx call.
  Add _is_retryable() filter so 404s don't burn 5 retry slots.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 23:49:35 +02:00
Tarik Moussa
b7f19b6e2c feat(sources): OpenAlex, SemanticScholar, arXiv API clients
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 23:31:15 +02:00