[MayCXC]

Ports asg017's experimental IVF index, the third of its approximate-nearest-neighbour indexes, as asg017's main has it at 04d28bd. This builds on #11 (DiskANN), which builds on #8: their five commits come first here, and the last two are this PR's.

What it adds. A build compiled with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 can declare a float vector column indexed by ivf(), which clusters its rows around nlist centroids (128 by default) with k-means and keeps them in cells of up to 64 rows, each cell holding rows of one centroid. A KNN query reads the cells of the nprobe centroids nearest its vector (10 by default) and the cells of the rows no clustering has placed yet, so until compute-centroids runs it reads every row. With quantizer=int8 or quantizer=binary, the cells and centroids hold the vectors quantized, the float vectors are kept by rowid besides, and a query ranks the k * oversample rows nearest by quantized distance by their exact distances.

create virtual table v using vec0(emb float[768] indexed by ivf(nlist=16, nprobe=4));
-- after the inserts
insert into v(v) values ('compute-centroids');
select rowid, distance from v where emb match ? and k = 10;
insert into v(v) values ('nprobe=8');  -- for this connection's queries

The commands compute-centroids, compute-centroids:{"nlist": N, "max_iterations": N, "seed": N}, set-centroid:N, assign-vectors, clear-centroids and nprobe=N build and tune the index. Its tables are _ivf_centroidsNN, _ivf_cellsNN, _ivf_rowid_mapNN and _ivf_vectorsNN, with the ivf_trained_N row of _info, named as upstream names them. The prebuilt binaries are built without it.

What it ports. asg017's sqlite-vec-ivf.c and sqlite-vec-ivf-kmeans.c as its main has them at 04d28bd (merged in e2c38f3), with the refusal of a binary quantizer on dimensions not divisible by 8 (4bee883), the refusal of a vector UPDATE (9df59b4), auxiliary columns (b7fc459), the command column (6e2c4c6), the rename of its shadow tables (8b81f40), the finalizing of every cached statement on commit (fe94171), and the fixes of the unmerged asg017#313: an insert takes the first free slot of a cell, where upstream takes the slot after its count and overwrites a live row once a delete leaves a hole; assign-vectors reads the centroids at the right stride and by their ids, which set-centroid can leave apart; clear-centroids and set-centroid keep a quantized column's cells and centroids quantized; and the int8 quantizer ranks by the column's metric, where upstream ranks by L2.

Where it differs from upstream.

• ivf_knn() takes rescore_knn()'s arguments and results: a rowid filter ranks the rows it names by their exact distances, distance constraints filter the rows the query read, and MMR reranks them. Upstream reads only the query vector and k, so it returns rows outside a rowid filter or a distance constraint.
• A quantized column's point query and full scan read its float vectors from _ivf_vectorsNN. Upstream returns the quantized bytes, and its point query reads a float vector's length past them.
• Whether a column has centroids is read from _info at each use. Upstream caches it per connection, so a connection that queried before another one ran compute-centroids reads only the cells of no centroid, which are gone, and finds no rows. Two connections to one file, 100 rows of float[8] in ivf(nlist=4, nprobe=4): a KNN on the first returns rows 7, 97 and 40; the second runs compute-centroids; the same KNN on the first then returns nothing on asg017's main, and 7, 97 and 40 here.
• The cells are found by centroid through the index of a UNIQUE (centroid_id, cell_id) constraint on a cell_id INTEGER PRIMARY KEY, which ALTER TABLE RENAME renames along with its table. Upstream creates an index named after the table, which keeps its name through a rename, so a table created later under the old name fails to create its own; nothing inside xRename can drop an index, since OP_Destroy refuses while the ALTER TABLE statement reads. Every statement on the cells names its columns and addresses a cell by rowid, so either implementation opens the other's tables.
• Only float columns can be indexed by ivf(). Upstream accepts int8 and bit columns and sizes their cells for float vectors, so it reads past an int8 vector and ranks a row's own vector at a distance such as 6.6e10.
• Errors reach the statement, so an interrupt stays an interrupt. Upstream ignores the results of its writes to the cells and of its blob reads and writes, and runs compute-centroids in a SAVEPOINT it opens inside xUpdate, which SQLite refuses while the statement writes. An interrupted command fails with interrupted and the statement's rollback undoes it; upstream reports SQL logic error, and the connection's KNN queries then find no rows. A cell or centroid blob whose size the column's options do not give fails with a message naming it.
• A cached statement is reset as soon as it has run, so VACUUM works after a KNN query.
• The commands apply to every IVF column of the table; upstream applies them to the first. compute-centroids:{...} is read with SQLite's json_each(), and its keys nlist, max_iterations and seed take integers in range; upstream finds them with strstr() and reads them with atoi(). The commands take digits only, and CREATE options and their values must be spelled in full; upstream accepts any prefix, as in quant=int8.
• k-means indexes its arrays with size_t; upstream multiplies int indexes, which overflow past 2^31 floats.
• optimize, which upstream lacks, leaves an IVF column's rows, which are keyed by rowid, as they are.

The IVF code shares the DiskANN code's helpers, which take names for both: diskann_stmt() and diskann_exec_rowid() become vec0_cached_stmt() and vec0_exec_i64(), and vec0_parse_diskann_int() becomes vec0_parse_option_int(). The commit adds an ### Added entry under [Unreleased] in CHANGELOG.md, and ARCHITECTURE.md and the README describe the index.

Tests.

• Fix ivf index bugs asg017/sqlite-vec#313's test-ivf.py, test-ivf-mutations.py, test-ivf-quantization.py and test-ivf-rename.py; upstream's eight IVF fuzz targets with their dictionary, built with the IVF index compiled in (upstream's fuzz Makefile builds them without it, so they create no IVF table); and the IVF cases of Fix ivf index bugs asg017/sqlite-vec#313's test-unit.c, which call the functions through tests/sqlite-vec-internal.h. Fix ivf index bugs asg017/sqlite-vec#313's test-ann-embeddings.py needs an embedding server on localhost and is not ported.
• New: a query reading every cell answers as a flat column does, constraint for constraint; the float vectors of a quantized column; a second connection after compute-centroids; a rename followed by a table under the old name; the refused columns and options; an interrupted command; the commands on every IVF column; the command errors; the compute-centroids options and seed; corrupt cells; INSERT OR REPLACE; VACUUM after a KNN query; optimize beside a flat column; and k-means in the unit tests.
• conftest.py skips the test-ivf files when vec_debug() names no IVF build flag, as upstream's does; make test-unit passes CPPFLAGS to the compiler, and a new CI job, build-linux-x86_64-ivf, compiles the IVF index in and runs every suite.
• The last commit adds three IVF tables to test_interrupt_is_reported_as_interrupt: KNN queries over the centroids, on a rowid filter and on a quantized column, full scans, a point query, inserts into cells of a centroid, of the quantized column and of no centroid, an auxiliary column update, deletes, optimize, compute-centroids with and without options, clear-centroids, set-centroid and assign-vectors, each interrupted at every progress handler call. A build whose writes to the cells return SQLITE_ERROR in place of vec0_error_rc()'s code fails 12 of the 36 cases.

Without the IVF index, make test-loadable passes (680 passed, 182 skipped); with CPPFLAGS=-DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 it passes 859 with 3 skipped, and so does the suite against an AddressSanitizer and UndefinedBehaviorSanitizer build. make test-unit and make test pass both ways. GCC 14.2 and Clang 19.1 with -Wall -Wextra print nothing for the loadable build with any combination of SQLITE_VEC_ENABLE_RESCORE, SQLITE_VEC_ENABLE_DISKANN and the IVF flag, and with -DSQLITE_CORE only main's three unused pApi warnings, which #9 removes. clang-format 21 leaves sqlite-vec.c unchanged, and before this rebase each of the eight IVF fuzz targets ran 90 seconds without a finding.

Rebased with #8 and #11 onto main at f6a3244, it merges cleanly with main and with #4. build-linux-x86_64-ivf follows #6's build-linux-aarch64-extension job in .github/workflows/test.yaml.

[mceachen]

We'll review the index ports one at a time once #4 and #7 are in, starting with rescore (#8).