Repository navigation
2/4: Warn whenever an out-of-memory layer is loaded into memory - #10495
Open
BenjaminDEMAILLE wants to merge 2 commits into
Open
BenjaminDEMAILLE wants to merge 2 commits into
BenjaminDEMAILLE wants to merge 2 commits into
Conversation
This was referenced Sep 1, 2026
BenjaminDEMAILLE
marked this pull request as ready for review
September 8, 2026 16:37
dgCMatrix indexes its non-zeros with 32-bit signed integers, so a matrix stops being representable at 2^31 - 1 entries: invalid class "dgCMatrix" object: 'i' slot is too large as.DelayedMatrix(x, ondisk = FALSE, file = NULL) wraps a matrix for use as a layer. In memory it gets an SVT_SparseMatrix seed, which stores non-zeros in a per-column tree rather than one flat index vector, so a layer can hold more than 2^31 of them and still serialize with saveRDS. On disk it is realized to a 10x-style TENxMatrix, whose indptr dataset is written with an 8-byte native HDF5 type; the test reads that type back rather than assuming it. Backend methods keep the standard pipeline block-wise, mirroring the existing IterableMatrix ones: LogNormalize.DelayedMatrix, VST.DelayedMatrix, .CalcN.DelayedMatrix, as.sparse.DelayedMatrix, and DelayedMatrix branches in RowMeanSparse, RowSumSparse and RowVarSparse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Operations with no block-wise form (CLR and RC normalization, mvp/dispersion feature selection) fall back to materializing the layer. That notice was a message() gated on `verbose`, so `verbose = FALSE` pulled the entire out-of-core matrix into RAM without saying anything. A user reaches for this backend precisely because the data may not fit in memory, so that is the one event they must never miss: at best it is a surprise memory spike, at worst the session dies. Make it an immediate warning that names the operation and the matrix dimensions, and points at the block-wise alternatives, with no way to silence it through `verbose`. Adds tests asserting the warning fires with verbose = FALSE, and that the block-wise paths stay quiet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BenjaminDEMAILLE
force-pushed
the
feat/delayedmatrix-warn-materialize
branch
from
September 8, 2026 16:47
5f9480f to
a351727
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of the series that replaces #10493. Depends on #10494 and is drafted until that lands; the diff below is what it adds on top.
59 lines, 3 files.
Some operations have no block-wise form: CLR and RC normalization, and
mvp/dispersionfeature selection. They load the layer into memory. Someone reaches for an out-of-memory backend precisely when the data may not fit in RAM, so this always emits awarning()naming the operation, the matrix dimensions and the block-wise alternative.Deliberately not gated on
verbose. I found this by testing my own first version, which usedmessage(): withverbose = FALSEit silently materialised a matrix that the user had specifically arranged not to hold in memory. A warning that cannot be switched off is the point of the change.Applies to BPCells layers as well, which have the same failure mode today.
Verified: 39 lines of tests asserting the warning fires under
verbose = FALSE, and that its text names the operation. Full suite: 1214 passing, the only failures being the two pre-existingRead10X_Imageones that #10492 fixes.🤖 Generated with Claude Code