Repository navigation
docs: render multiple_slides with the others and ship a 1.2 GB stomach dataset - #9
Merged
Merged
Conversation
The same dask code now renders with the other tutorials on the CPU runner, on three small slides, and the 45-slide result comes from LazySlide-data, as genomics_integration already does.
The balanced sampler only ever used 38,500 of the 1.5M tiles in the 15.4 GB zip. scripts/build_stomach_compact.py records how the compact zip was built.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
It runs the multiple_slides steps on all 45 artery slides. [skip ci]
agg_conch_features.h5ad on LazySlide-data (dataset commit 88bda71c) now comes from scripts/build_agg_conch_features.py, so its rows for the demo slides equal the notebook's live scores. genomics_integration shows the table it loads, which also makes CI re-run it with the new file.
The cache entry (26 GB on this PR) exceeded GitHub's 10 GB per-repo cache limit, so GitHub evicted it right after the save and no run ever restored it. Every render downloaded the files fresh (about 2.5 min) and then spent about 75 s uploading a cache that did not survive. The offline check only verified that cache, so it goes too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR C of the tutorial plan. Two tutorials that needed a GPU cluster or a 15 GB download now render with the others on CPU. Task C2 of the plan (DeepSpot-M on a subset of spots) already shipped in #8.
multiple_slides, moved from
tutorials/heavy/totutorials/LocalCUDAClusterfrom dask-cuda on a machine with GPUs.agg_conch_features.h5adfor all 45 slides from LazySlide-data, asgenomics_integrationalready does, and shows that its rows for the demo slides equal the live scores.scripts/setup_env.pyprefetches the three slides and the slide table.render_heavy.yamlnow requires a notebook name, sincetutorials/heavy/is empty.multiple_slidesandbuild_tile_modelexecuted (the others came from the cache), the whole notebook took 5.5 min. A cold render of all notebooks could take 16 to 25 min, still under the 30-min cell limit.agg_conch_features.h5ad, regenerated
scripts/build_agg_conch_features.pyruns the notebook's per-slide steps on all 45 slides and writes the table. The old file came from an earlier version of the code that averaged unit-normalized tile features, so its scores differed from the tutorial's.genomics_integrationnow displays the table it loads. That also makes CI re-run it with the new file, since jupyter-cache re-runs a notebook only when its code changes.build_tile_model
gtex_stomach_subset_compact.zip(1.2 GB) instead ofgtex_stomach_subset.zip(15.4 GB). It holds 7,700 training tiles of each class, the size of the smallest class, plus the one test slide the tutorial plots.scripts/build_stomach_compact.pyrecords how the compact zip was built, with a fixed seed. Test accuracy is 0.796 on the compact data, against 0.798 on the full data.CI
build_notebooks.yaml. The cache entry (26 GB here) exceeded GitHub's 10 GB per-repo cache limit, so GitHub evicted it after every save and no run ever restored it. Downloading takes about 2.5 min, and the save cost about 75 s per run for nothing. The offline check only verified that cache, so it goes too.Checks
multiple_slidesandgenomics_integrationran end to end locally against the new file with no errors.