Repository navigation
3/4: Convert on-disk assays when building a Seurat object from an SCE - #10496
Open
BenjaminDEMAILLE wants to merge 3 commits into
Open
BenjaminDEMAILLE wants to merge 3 commits into
BenjaminDEMAILLE wants to merge 3 commits into
Conversation
BenjaminDEMAILLE
marked this pull request as ready for review
September 8, 2026 16:37
dgCMatrix indexes its non-zeros with 32-bit signed integers, so a matrix stops being representable at 2^31 - 1 entries: invalid class "dgCMatrix" object: 'i' slot is too large as.DelayedMatrix(x, ondisk = FALSE, file = NULL) wraps a matrix for use as a layer. In memory it gets an SVT_SparseMatrix seed, which stores non-zeros in a per-column tree rather than one flat index vector, so a layer can hold more than 2^31 of them and still serialize with saveRDS. On disk it is realized to a 10x-style TENxMatrix, whose indptr dataset is written with an 8-byte native HDF5 type; the test reads that type back rather than assuming it. Backend methods keep the standard pipeline block-wise, mirroring the existing IterableMatrix ones: LogNormalize.DelayedMatrix, VST.DelayedMatrix, .CalcN.DelayedMatrix, as.sparse.DelayedMatrix, and DelayedMatrix branches in RowMeanSparse, RowSumSparse and RowVarSparse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Operations with no block-wise form (CLR and RC normalization, mvp/dispersion feature selection) fall back to materializing the layer. That notice was a message() gated on `verbose`, so `verbose = FALSE` pulled the entire out-of-core matrix into RAM without saying anything. A user reaches for this backend precisely because the data may not fit in memory, so that is the one event they must never miss: at best it is a surprise memory spike, at worst the session dies. Make it an immediate warning that names the operation and the matrix dimensions, and points at the block-wise alternatives, with no way to silence it through `verbose`. Adds tests asserting the warning fires with verbose = FALSE, and that the block-wise paths stay quiet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
as.Seurat() on a SingleCellExperiment whose assays are on-disk failed with an opaque S4 error: assignment of an object of class "RenameDims" is not valid for slot 'data' in an object of class "Assay" A v3 Assay keeps its matrices in slots typed for in-memory classes, so an IterableMatrix or DelayedMatrix can never be stored there. Materialize explicitly at that boundary, through the same warning used elsewhere, so the user is told what happened and what it cost instead of reading a class error. This also fixes the equivalent pre-existing failure for BPCells. Verified against dgCMatrix, DelayedMatrix and BPCells: all three now round -trip through as.SingleCellExperiment()/as.Seurat() with identical values, and only the on-disk ones warn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BenjaminDEMAILLE
force-pushed
the
feat/delayedmatrix-sce
branch
from
September 8, 2026 16:47
42f11cc to
fdf6176
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of the series that replaces #10493. Depends on #10494 and #10495 and is drafted until those land.
64 lines, 5 files.
as.Seurat()on aSingleCellExperimentwhose assays are out-of-memory produced an object holding matrices that the rest of the pipeline could not use. It now converts on-disk assays as it builds the object, so aSingleCellExperimentcarrying aDelayedMatrixbecomes a Seurat object with a working layer.Verified: 31 lines of tests round-tripping an SCE with an out-of-memory assay. Full suite: 1218 passing, the only failures being the two pre-existing
Read10X_Imageones that #10492 fixes.🤖 Generated with Claude Code