Skip to content

Re-collect every faculty member's publications in scheduled refreshes - #7

Merged
gvegayon merged 1 commit into
mainfrom
feat/weekly-faculty-publications
Oct 22, 2026
Merged

gvegayon merged 1 commit into
mainfrom
feat/weekly-faculty-publications

Conversation

@gvegayon

Copy link
Copy Markdown
Member

Why

The weekly (roster) and odd-month (profile) refreshes only re-queried publications for faculty whose publication inputs (ORCID, PubMed/arXiv query, collect flag) changed. Everyone else's new papers never arrived — e.g. PMID 42853165 (Sideris with Nelson, Zickmund, Stehlik, Hanff) was missing although all four faculty queries return it.

What

  • doim-refresh --roster-only / --all-profiles now target every faculty member with publication collection enabled. Targeted --faculty-ids runs are unchanged.
  • Failure policy: errors for faculty whose inputs changed stay fatal; for unchanged faculty an error keeps their previous works (history merge) and the run fails only if more than 10% of them err (MAX_UNCHANGED_ATTENTION_SHARE), i.e. a source outage.
  • RAG stays incremental: only new chunks are embedded.
  • README and migration plan (new unchecked P7.4) updated.

Cost

Local full pull on 2026-10-21: 12 min for 508 faculty, 7,288 → 7,347 works (59 new, including PMID 42853165), 1 faculty source errored. The job timeout is 45 min.

Test plan

  • uv run --locked pytest (247 passed), ruff check
  • New tests: a scheduled refresh re-collects an unchanged faculty member; an outage fails the run; an error below the threshold keeps previous works
  • First scheduled run completes within the timeout

Independent of #6; whichever merges second needs no conflict resolution beyond docs/doim-migration-plan.md.

🤖 Generated with Claude Code

…reshes

Weekly and odd-month refreshes queried publication sources only for
faculty whose publication inputs changed, so new papers by everyone else
never arrived. Scheduled runs now target every faculty member with
collection enabled. Source errors stay fatal for changed faculty; for the
rest an error keeps their previous works and fails the run only above
10% (an outage). A local full pull took 12 minutes for 508 faculty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gvegayon
gvegayon merged commit c797ae7 into main Oct 22, 2026
1 check passed
@gvegayon
gvegayon deleted the feat/weekly-faculty-publications branch October 22, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant