Skip to content

feat: improve search reliability and add latest dataset update tracking - #146

Merged
elainefan331 merged 18 commits into
stagingfrom
dev-fan
Sep 10, 2026
Merged

feat: improve search reliability and add latest dataset update tracking#146
elainefan331 merged 18 commits into
stagingfrom
dev-fan

Conversation

@elainefan331

Copy link
Copy Markdown
Collaborator

Summary

This PR improves NeuroJSON search behavior and adds persistent sync/update tracking for the landing page.

It includes two search fixes, PostgreSQL-backed statistics snapshots, dataset-level incremental change tracking, a latest-update API, and the new LatestUpdateBoard UI.

What changed

Search

  • Fix dataset-level keyword matching when subject filters are active by checking the corresponding dbinfo row.
  • Prevent the false “No matching subjects” warning from flashing during slow searches.
  • Add an index on iolinks(dbname, dsname, view) to improve file-type filtering performance.

Statistics

  • Move landing-page database statistics from the legacy CGI path to PostgreSQL.
  • Add stats_history to persist one statistics snapshot per sync run.
  • Update the sync pipeline to write dataset, subject, file, and size totals after successful syncs.
  • Serve landing-page statistics from the latest successful stats_history snapshot.

Dataset change tracking

  • Add dataset_changes to track dataset-level added, updated, and deleted events during incremental syncs.
  • Link each change to its corresponding stats_history run.
  • Keep firstSync() excluded from change logging.

Latest update API

  • Add GET /dbs/updates/latest.
  • Return the most recent successful sync that contains actual dataset changes.
  • Include:
    • added / updated / deleted dataset counts
    • affected datasets
    • net subject, file, and size deltas based on stats_history snapshots

Landing page

  • Add fetchLatestUpdate service / Redux action / slice support.
  • Add LatestUpdateBoard to the landing page.
  • Display the latest update date, dataset changes, subject/file/size deltas, and affected datasets.
  • Refine typography, colors, spacing, and alignment to match the existing NeuroJSON theme.

Testing

  • Verified keyword + subject-filter searches.
  • Verified slow-search warning behavior.
  • Verified incremental dataset add / update / delete tracking.
  • Verified dataset_changes rows are linked to the correct sync history.
  • Verified the latest-update endpoint response.
  • Verified landing-page statistics and latest-update rendering.

Related issue

Closes #145

@elainefan331
elainefan331 merged commit aff4363 into staging Sep 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add dataset change tracking and latest-update summary to landing page

1 participant