[BenjaminDEMAILLE]

First of a small series that replaces #10493, split so each piece can be reviewed on its own. Tracking issue with the order and the dependencies: #10498

This PR: 522 lines, 11 files, one idea. It is the whole of the 2^31 fix; the rest of the series is optional on top of it.

Why

dgCMatrix indexes its non-zeros with 32-bit signed integers, so a matrix stops being representable at 2^31 - 1 entries:

invalid class "dgCMatrix" object: 'i' slot is too large (> 2^31 - 1 elements)

Real data already crosses that line: the 10x GEM-X Flex 1M human K562 CRISPR aggregate has 4,355,024,689 non-zeros. Today BPCells is the only way past it, which means a dependency, a specific on-disk format, and objects that stop working when their directory moves.

What it adds

as.DelayedMatrix(x, ondisk = FALSE, file = NULL) wraps a matrix for use as a layer:

• in memory, sparse input gets an SVT_SparseMatrix seed (SparseArray), which stores non-zeros in a per-column tree rather than one flat index vector, so a layer can hold more than 2^31 of them and still serialize with saveRDS;
• on disk (ondisk = TRUE), it is realized to a 10x-style TENxMatrix, whose indptr dataset is written with an 8-byte native HDF5 type, so the same limit is lifted while the data stays out of memory.

Backend methods keep the standard pipeline block-wise, mirroring the existing IterableMatrix ones: LogNormalize.DelayedMatrix, VST.DelayedMatrix, .CalcN.DelayedMatrix, as.sparse.DelayedMatrix, and DelayedMatrix branches in RowMeanSparse, RowSumSparse and RowVarSparse.

What it deliberately leaves out

Everything separable, so this PR stays one idea: the materialization warning, the SingleCellExperiment conversion, the CheckMatrixSize reporter, and object portability are each their own PR in the series. None of them is needed for this one to be useful or correct.

Verification

168 lines of tests, including: a matrix round-trips through the backend unchanged; LogNormalize and VST give the same numbers as the dgCMatrix path; the on-disk form's indptr type is read back from the written file and asserted to be 8 bytes, rather than assumed. Full suite: 1209 passing, the only failures being the two pre-existing Read10X_Image ones that #10492 fixes.

🤖 Generated with Claude Code