Skip to content

Build the fuzz targets with the experimental IVF index - #355

Open
MayCXC wants to merge 3 commits into
asg017:mainfrom
MayCXC:fuzz-ivf-define
Open

MayCXC wants to merge 3 commits into
asg017:mainfrom
MayCXC:fuzz-ivf-define

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 23, 2026 •

Copy link
Copy Markdown

The fuzz Makefile builds its eight IVF targets without SQLITE_VEC_EXPERIMENTAL_IVF_ENABLE, so the library they link has no IVF index: each create virtual table ... indexed by ivf(...) they run fails to parse, and no IVF code runs. bb3ef78 replaced SQLITE_VEC_ENABLE_IVF, which defaulted to 1, with SQLITE_VEC_EXPERIMENTAL_IVF_ENABLE, which defaults to 0, and the fuzz build kept the default. FUZZ_CFLAGS now defines it beside SQLITE_VEC_ENABLE_DISKANN, so every target builds with the IVF index, as every target did before bb3ef78.

This builds on #343, without which no fuzz target builds: its two commits come first here, and the last one is this PR's.

Test. With clang 19 on Linux arm64, from tests/fuzz,

./targets/ivf_create -seed=1 -runs=5000 -dict=ivf-create.dict -print_coverage=1

covers 690 functions on #343's branch, none of them IVF's, and 875 here, vec0_parse_ivf_options() and ivf_create_shadow_tables() among them. All eight IVF targets build. With the index compiled in, ivf_cell_overflow finds an 8-byte input that UBSan reports as a misaligned float load:

$ printf '\377\377\377\377\377\377\236\000' > crash
$ ./targets/ivf_cell_overflow crash
../../sqlite-vec.c:390:23: runtime error: load of misaligned address 0x50e00001e971 for type 'f32' (aka 'float'), which requires 4 byte alignment
    #0 in l2_sqr_float ../../sqlite-vec.c:390:23
    ...
    #4 in ivf_find_nearest_centroid ../../sqlite-vec-ivf.c:220:18
    #5 in ivf_cmd_assign_vectors ../../sqlite-vec-ivf.c:1107:17

assign-vectors passes the dimension count where ivf_find_nearest_centroid() takes the size of a vector in bytes, so it reads centroid c from byte c * D instead of byte 4 * c * D, inside the centroids before it. #313 passes the byte size there, and with #313 merged the input runs clean.

Origin. The IVF port to photostructure/sqlite-vec, photostructure#12 (65817f4), found that these targets build without the index, and passes the define to the eight IVF targets alone. Here it goes in FUZZ_CFLAGS, the build every target had before bb3ef78. No open or closed PR or issue here builds the fuzz targets with it; stumpylog's #305 has tests/test-unit.c follow the same rename. Besides #343, the open PR that touches tests/fuzz is #269, which conflicts with main in tests/fuzz/Makefile with or without this change.

575371d, which added the DiskANN targets, dropped the line continuation
after ivf_rescore in FUZZ_TARGETS and the ivf_rescore rule's recipe, so
make stops at the first DiskANN name and builds nothing:

    $ make -C tests/fuzz all
    Makefile:158: *** missing separator (did you mean TAB instead of 8
    spaces?).  Stop.

0de765f then removed numpy.c, which the numpy target still compiles. The
list continues past ivf_rescore again, ivf_rescore has its recipe, and the
numpy target and its dictionary are gone. make -C tests/fuzz all builds all
36 targets.
575371d replaced rescore's min/max int8 quantizer with a fixed map of
[-1, 1] onto [-128, 127], and mode 3 still asserted that a vector of one
repeated value quantizes to all zeros, as the replaced quantizer did for a
zero range. The current quantizer gives 0 only for values near 0.004, so
the target aborts within a few hundred inputs (a vector of NaN gives 127).
Mode 3 now asserts that every element gets the same code, which holds for
any element-wise quantizer, and the comments on modes 2 and 4 describe the
clamps the current quantizer has.

Modes 1 and 2 read the fuzz bytes as floats from one byte past the mode
byte, a misaligned load UBSan reports in rescore_quantize_float_to_int8();
they copy the bytes to a float array first. The target runs 3.7 million
inputs in 60 seconds without a report.
bb3ef78 made the IVF index opt-in behind
SQLITE_VEC_EXPERIMENTAL_IVF_ENABLE, off by default, and the fuzz
Makefile kept relying on the old default, so its eight IVF targets have
built without the index since and fuzz a CREATE that cannot parse
`indexed by ivf(...)`. FUZZ_CFLAGS now defines it beside
SQLITE_VEC_ENABLE_DISKANN, which gives every target the IVF index the
library compiled in before bb3ef78.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant