user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
15 Commits 5 Branches 0 Tags
1e1073618572e381377aa52f61ccf1a03b358879
Commit Graph

11 Commits

Author SHA1 Message Date
Tarik Moussa
5344975bb1 feat(ingest): end-to-end idempotent ingest pipeline 
Implements codex/ingest.py with:
- OpenAlex primary / Semantic Scholar fallback metadata fetch
- Abstract dense embedding (zero-vector for missing abstracts)
- Idempotent paper upsert (ON CONFLICT DO UPDATE)
- Source file parsing (.tex via latex_to_text, .pdf via nougat + grobid)
- Chunk deletion + bulk re-insert with dense embeddings
- Citation merge from OpenAlex/S2 API and GROBID PDF refs (dedup via set)
- 7 tests covering basic, tex, pdf, not-found, idempotent, arxiv-fallback,
  and no-abstract scenarios (all mocked, offline)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-05 07:04:12 +02:00
Tarik Moussa
fbf8949579 feat: merge F-04 embed layer (BGE-M3 dense+sparse via FlagEmbedding) 2026-06-05 06:56:17 +02:00
Tarik Moussa
c33fd9d697 feat: merge F-03 parsing layer (LaTeX, Nougat, GROBID) 
Review-Gate: APPROVE (Opus, cold, 2 passes)
23 tests, ruff+mypy clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-05 06:48:49 +02:00
Tarik Moussa
52737abe3d fix(sources): correct OpenAlex ID prefixing, citations endpoint, S2 rate-limit 
Review-Gate findings:
- openalex: bare DOIs/arXiv IDs need doi:/arxiv: prefix (bare IDs 404);
  add _resolve_id(); fix fetch_citations to use referenced_works field
  instead of non-existent /works/{id}/references endpoint.
- semanticscholar: wait_fixed(1) was inter-retry only; add per-request
  monotonic rate-limiter (_rate_limit()) before each httpx call.
  Add _is_retryable() filter so 404s don't burn 5 retry slots.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 23:49:35 +02:00
Tarik Moussa
1698f7dcad feat(embed): BGE-M3 dense+sparse via FlagEmbedding (ADR-0002) 
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-04 23:48:06 +02:00
Tarik Moussa
d1ffea1c1c fix(parsing): case-insensitive arXiv idno match; fix tmp_path types; add extract_structure test 
Review-Gate finding: GROBID emits type="arXiv" (camel-case), not "arxiv".
XPath literal match silently returned empty strings for all real responses.
Fix: iterate idno elements and compare .lower() == "arxiv".
Test fixture updated to reflect real GROBID output (type="arXiv").

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 23:47:18 +02:00
Tarik Moussa
b7f19b6e2c feat(sources): OpenAlex, SemanticScholar, arXiv API clients 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 23:31:15 +02:00
Tarik Moussa
c6a428d335 feat(parsing): LaTeX, Nougat, GROBID parsers with chunking 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 23:30:58 +02:00
Tarik Moussa
b52a6a7412 fix: add @app.callback so codex --help exits 0 
Bare Typer() with no commands raised RuntimeError at runtime.
Callback makes the entry point callable without exposing F-07 logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 11:26:35 +02:00
Tarik Moussa
1dd7a4bea3 fix: add cli.py stub so entry point resolves before F-07 
Review-Gate finding: pyproject.toml [project.scripts] referenced
codex.cli:app which was absent; uv run codex raised ModuleNotFoundError.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 11:25:17 +02:00
Tarik Moussa
e7ac7766a5 feat: initial project scaffold 
pyproject.toml (Python 3.12, uv), codex/ package (config, db, models),
infra/ (docker-compose + schema), .env.example, .gitignore, README.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 11:19:10 +02:00