user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ingest): support .txt sources + fix citing_id FK violation #7

Merged
user2595 merged 1 commits from fix/ingest-txt-and-citing-id into main 2026-06-14 13:56:05 +00:00
Conversation 0 Commits 1 Files Changed 1 +6 -1
user2595 commented 2026-06-14 13:54:21 +00:00
Owner
Copy Link

Summary

Two bugs found during the first full batch ingest run (29 papers → Jetson DB via SSH tunnel):

Bug 1 — .txt files silently skipped

  • ingest.py only handled .tex and .pdf extensions; .txt fell through to logger.warning("Unbekannter Dateityp") and produced 0 chunks
  • Fix: added elif suffix == ".txt": text = Path(source_path).read_text(...) branch

Bug 2 — citing_id FK violation after D-03

  • Before D-03: OpenAlex 404d on arXiv IDs → openalex_id=None → SemanticScholar fallback → S2 used arXiv ID as citing_id = paper.id ✓
  • After D-03: OpenAlex succeeds → paper.id = 'https://doi.org/…' but fetch_citations() returns citing_id = openalex_id = 'https://openalex.org/W…' → FK constraint citations_citing_id_fkey → papers.id violated
  • Fix: rewrite citing_id to paper.id after fetching from OpenAlex

Test plan

  • uv run pytest tests/ingest/ passes
  • Ingest a paper with .txt source → chunks > 0
  • Ingest a paper that OpenAlex resolves → no FK violation on citations
## Summary Two bugs found during the first full batch ingest run (29 papers → Jetson DB via SSH tunnel): **Bug 1 — `.txt` files silently skipped** - `ingest.py` only handled `.tex` and `.pdf` extensions; `.txt` fell through to `logger.warning("Unbekannter Dateityp")` and produced 0 chunks - Fix: added `elif suffix == ".txt": text = Path(source_path).read_text(...)` branch **Bug 2 — `citing_id` FK violation after D-03** - Before D-03: OpenAlex 404d on arXiv IDs → `openalex_id=None` → SemanticScholar fallback → S2 used arXiv ID as `citing_id` = `paper.id` ✓ - After D-03: OpenAlex succeeds → `paper.id = 'https://doi.org/…'` but `fetch_citations()` returns `citing_id = openalex_id = 'https://openalex.org/W…'` → FK constraint `citations_citing_id_fkey → papers.id` violated - Fix: rewrite `citing_id` to `paper.id` after fetching from OpenAlex ## Test plan - [ ] `uv run pytest tests/ingest/` passes - [ ] Ingest a paper with `.txt` source → chunks > 0 - [ ] Ingest a paper that OpenAlex resolves → no FK violation on citations
user2595 added 1 commit 2026-06-14 13:54:22 +00:00
fix(ingest): support .txt sources + fix citing_id FK violation 5025ff3a88 
Two bugs exposed during first batch ingest run after D-03 fix:

1. `.txt` files were silently skipped ("Unbekannter Dateityp") — added
   plain-text read path alongside .tex and .pdf.

2. After D-03, OpenAlex resolves arXiv IDs → paper.id uses the DOI
   (https://doi.org/…) but fetch_citations emits citing_id=openalex_id
   (https://openalex.org/W…). FK constraint citations_citing_id_fkey
   → papers.id violated. Fix: rewrite citing_id to paper.id after
   fetching from OpenAlex.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
user2595 merged commit 1bec18ae82 into main 2026-06-14 13:56:05 +00:00
user2595 deleted branch fix/ingest-txt-and-citing-id 2026-06-14 13:56:05 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
user2595
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: user2595/codex-py#7
No description provided.
Delete Branch "fix/ingest-txt-and-citing-id"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?