Research corpus — literature ingestion

Getting a paper from arXiv into the canonical, citable knowledge base.

Stream: research-corpus · closed 2026-07-03

The question

  • How does a paper get from arXiv into the citable corpus?
  • "Done" = filed under raw/md/<bibkey>/ + raw/references.bib + a wiki source page.
  • This stream ran that path end-to-end for the ICM paper (vanclief2026interpretable).

How we closed it — convert, then promote

  • Convert: arxiv-to-prompt → pandoc -f latex -t markdown --wrap=none.
  • Stage under a new context/ bucket, commit 13b53abc.
  • Promote: file raw/md/vanclief2026interpretable/, append raw/references.bib.
  • Wrote the wiki source page (review: verified), indexed + logged.
  • references.bib verified against Semantic Scholar (arXiv-only, no journal DOI).

How we closed it — the out-of-domain tag

  • The ICM paper is a methodology paper — no closed-vocab tag fit.
  • Flagged PAGE NEEDED: topics/context_engineering; logged for user OK.
  • Decision 2026-07-02: added topics/context_engineering + a stub topic page.
  • Retagged the source page, index 94→95, shipped 0296167e.

How we closed it — research-wiki equations

  • Swept ~20 vault/research pages (notation, sources, topics, results).
  • Pandoc \(...\)/\[...\] → Obsidian $...$/$$...$$; unicode math → LaTeX.
  • Math verified preserved verbatim on sampled diffs (das2025understanding line-by-line).
  • The regenerated lint report is now the wiki-health source of truth.

What we learned

  • bibkey is the join key: wiki page ↔ .bib ↔ \cite.
  • A closed tag vocabulary extends deliberately (user OK), never force-fit.
  • arXiv-only preprints carry no journal DOI — verify the bib elsewhere.
  • Pandoc's math delimiters don't render in Obsidian — convert on ingest.
  • A lost structured-output count → regenerate the check, don't cite memory.

Side effects

  • New context/ staging bucket joins CVaR and nominal.
  • Equations sweep touched ~20 wiki pages — wider than one paper's ingest.
  • Fixed post-reorg stale paths in the arxiv-lookup skill + allowlist.
  • Added a bib entry-format note to research-assistant/SKILL.md.

What comes next

  • topics/context_engineering is a STUB — flesh it out when the topic recurs.
  • Confirm no pandoc-delimiter stragglers remain, against the regenerated lint report.
  • No PDF for this preprint — add a DOI + PDF if a journal version lands.

Terminology

No term this stream touched has a citation in notes/terminology.md.

  • context_engineering is a research-wiki TAG, not a sourced science term.
  • bibkey and raw/ are infra identifiers, not sourced science terms.

So no term earns a slide (precedent: the facade-retirement deck).