Skip to content

docs: render multiple_slides with the others and ship a 1.2 GB stomach dataset - #9

Merged
Mr-Milk merged 7 commits into
mainfrom
docs/light-heavy-tutorials
Oct 6, 2026
Merged

Mr-Milk merged 7 commits into
mainfrom
docs/light-heavy-tutorials

Conversation

@Mr-Milk

@Mr-Milk Mr-Milk commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

PR C of the tutorial plan. Two tutorials that needed a GPU cluster or a 15 GB download now render with the others on CPU. Task C2 of the plan (DeepSpot-M on a subset of spots) already shipped in #8.

multiple_slides, moved from tutorials/heavy/ to tutorials/

  • Processes three small slides live (218, 292 and 298 tiles) on a local dask cluster with three workers, one per slide. The text says to use LocalCUDACluster from dask-cuda on a machine with GPUs.
  • Shows the live scores of the three demo slides, then loads agg_conch_features.h5ad for all 45 slides from LazySlide-data, as genomics_integration already does, and shows that its rows for the demo slides equal the live scores.
  • In the 45-slide table, calcified slides score higher on calcification (0.564 against 0.276).
  • scripts/setup_env.py prefetches the three slides and the slide table. render_heavy.yaml now requires a notebook name, since tutorials/heavy/ is empty.
  • Locally the dask cell takes 55 to 80 s on the GPU and about 100 s on the CPU. CI runs each dask worker on one CPU thread. In this PR's first render, where only multiple_slides and build_tile_model executed (the others came from the cache), the whole notebook took 5.5 min. A cold render of all notebooks could take 16 to 25 min, still under the 30-min cell limit.

agg_conch_features.h5ad, regenerated

  • scripts/build_agg_conch_features.py runs the notebook's per-slide steps on all 45 slides and writes the table. The old file came from an earlier version of the code that averaged unit-normalized tile features, so its scores differed from the tutorial's.
  • Built on HF Jobs under RendeiroLab (one L4, 36 min, about $0.50) and uploaded to LazySlide-data as commit 88bda71c, replacing the old file. The Hub history keeps the old one. The rows, columns and slide order are unchanged, and the demo slides' rows equal the live scores to within 5e-7.
  • genomics_integration now displays the table it loads. That also makes CI re-run it with the new file, since jupyter-cache re-runs a notebook only when its code changes.

build_tile_model

  • Downloads gtex_stomach_subset_compact.zip (1.2 GB) instead of gtex_stomach_subset.zip (15.4 GB). It holds 7,700 training tiles of each class, the size of the smallest class, plus the one test slide the tutorial plots.
  • scripts/build_stomach_compact.py records how the compact zip was built, with a fixed seed. Test accuracy is 0.796 on the compact data, against 0.798 on the full data.
  • The compact zip is on RendeiroLab/LazySlide-data (dataset commit cd4114bc). The old zip stays there for older versions of the tutorial.

CI

  • Drops the HF download cache from build_notebooks.yaml. The cache entry (26 GB here) exceeded GitHub's 10 GB per-repo cache limit, so GitHub evicted it after every save and no run ever restored it. Downloading takes about 2.5 min, and the save cost about 75 s per run for nothing. The offline check only verified that cache, so it goes too.

Checks

  • Both notebooks ran end to end locally with no errors, and the edited code passes ruff 0.11.12.
  • The notebook downloaded the compact zip from the Hub in that run, and its checksum matches the uploaded file.
  • After the regeneration, multiple_slides and genomics_integration ran end to end locally against the new file with no errors.

The same dask code now renders with the other tutorials on the CPU runner, on three small slides, and the 45-slide result comes from LazySlide-data, as genomics_integration already does.
The balanced sampler only ever used 38,500 of the 1.5M tiles in the 15.4 GB zip. scripts/build_stomach_compact.py records how the compact zip was built.
Copilot AI balanced review requested due to automatic review settings October 5, 2026 18:45
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Mr-Milk and others added 5 commits October 5, 2026 18:58
It runs the multiple_slides steps on all 45 artery slides. [skip ci]
agg_conch_features.h5ad on LazySlide-data (dataset commit 88bda71c) now comes from scripts/build_agg_conch_features.py, so its rows for the demo slides equal the notebook's live scores. genomics_integration shows the table it loads, which also makes CI re-run it with the new file.
The cache entry (26 GB on this PR) exceeded GitHub's 10 GB per-repo cache limit, so GitHub evicted it right after the save and no run ever restored it. Every render downloaded the files fresh (about 2.5 min) and then spent about 75 s uploading a cache that did not survive. The offline check only verified that cache, so it goes too.
@Mr-Milk
Mr-Milk merged commit 028370c into main Oct 6, 2026
1 check passed
@Mr-Milk
Mr-Milk deleted the docs/light-heavy-tutorials branch October 6, 2026 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants