PiPNN 5/6: add dedicated benchmark pipelines

[weiyaoluo (SeliMeli)]

Purpose

This PR adds dedicated benchmark routes for PiPNN graph and disk builds.

PiPNN builds adjacency in one batch. The benchmark must not use the incremental Vamana insertion path.

Main changes

• Graph inputs accept BuildAlgorithm::PiPNN.
• pipnn_build creates the requested Rayon pool and calls the PiPNN core.
• The graph route installs vectors, adjacency rows, and frozen start points after the batch build.
• The disk route calls the production builder from PiPNN 4/6: integrate disk build pipeline #1291.
• Feature-disabled PiPNN requests fail with an error.
• Dynamic graph jobs reject the batch-only algorithm.
• Example JSON files cover graph and disk routes.

BuildStats reports one batch duration. It does not report synthetic per-insert latency.

Review order

1. Review PiPNN input parsing in inputs/graph_index.rs and inputs/disk.rs.
2. Review index/build.rs::pipnn_build in execution order.
3. Review route selection in index/benchmarks.rs.
4. Review the disk route into diskann-disk.
5. Review example inputs and CLI tests.

Validation

• Tests check the requested start strategy and frozen adjacency.
• Tests check that batch builds omit insert percentiles.
• Tests check feature-disabled and dynamic-job errors.
• CLI tests cover graph and disk dispatch.
• All-target Clippy passes.

Stack

Stack 5/6. Depends on #1291. #1295 adds optional HashPrune merging.

[Copilot]

🟡 Not ready to approve

The pipnn feature definition likely does not enable the optional diskann-disk dependency (breaking --features pipnn builds) and should be corrected before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Adds PiPNN-specific benchmark entry points and fixtures so benchmarks can build PiPNN graphs (in-memory) and PiPNN disk indexes through the production disk builder, with integration tests validating the selected algorithm and basic results shape.

Changes:

• Add a dedicated in-memory PiPNN graph build pipeline in diskann-benchmark and route graph-index benchmarks to it when requested.
• Extend disk-index benchmark input/configuration to support selecting BuildAlgorithm (Vamana vs PiPNN), and pass PiPNN through the production disk build pipeline.
• Add PiPNN benchmark example JSONs and integration tests gated on the pipnn feature.

File summaries

File	Description
diskann-benchmark/src/main.rs	Extends CLI integration tests to cover PiPNN graph/disk benchmark examples.
diskann-benchmark/src/inputs/graph_index.rs	Adds optional PiPNN build algorithm selection for graph-index builds (feature-gated).
diskann-benchmark/src/inputs/disk.rs	Adds alpha, makes quantization optional, and adds build-algorithm selection/validation for disk-index builds.
diskann-benchmark/src/index/build.rs	Implements the dedicated PiPNN in-memory build pipeline and a unit test for start strategy handling.
diskann-benchmark/src/index/benchmarks.rs	Dispatches graph-index builds to PiPNN vs incremental insertion based on requested algorithm.
diskann-benchmark/src/disk_index/build.rs	Plumbs BuildAlgorithm into disk index build parameters (Vamana vs PiPNN).
diskann-benchmark/example/pipnn-graph-index.json	New example config exercising PiPNN graph-index build + TopK search.
diskann-benchmark/example/pipnn-disk-index.json	New example config exercising PiPNN disk-index build + search.
diskann-benchmark/Cargo.toml	Adds the pipnn feature and optional dependency on diskann-pipnn.
Cargo.lock	Adds diskann-pipnn to the workspace lockfile dependency graph.

Review details

• Files reviewed: 9/10 changed files
• Comments generated: 2
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

The PiPNN benchmark path has at least one confirmed config/behavior mismatch (silently ignoring multi_insert) and an avoidable performance issue in start-point source mapping that can skew benchmark timings.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (2)

diskann-benchmark/src/index/build.rs:160

• The PiPNN start-point source selection scans the dataset twice when an exact byte match is not found (position then min_by). For large start_point_strategy sample counts this adds avoidable O(2*N) work per start vector and skews the benchmark timing. Consider doing a single pass that checks for an exact match and otherwise tracks the best (minimum) distance as you iterate once.

    let start_sources = start_points
        .row_iter()
        .map(|start| {
            let bytes: &[u8] = bytemuck::cast_slice(start);
            data.row_iter()

diskann-benchmark/src/index/benchmarks.rs:240

• When build_algorithm is set to PiPNN, the benchmark bypasses the incremental builder and ignores multi_insert if it was provided in the input. This can silently mislead users into thinking multi-insert settings are applied to PiPNN builds. It would be safer to reject multi_insert for PiPNN with a clear error.

                let result = match build.build_algorithm() {
                    diskann_disk::BuildAlgorithm::PiPNN(parameters) => {
                        let data =
                            Arc::new(datafiles::load_dataset(datafiles::BinFile(build.data()))?);
                        build::pipnn_build(data, build, parameters)

• Files reviewed: 9/10 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

The pipnn feature configuration and PiPNN dispatch path have correctness issues that can cause build failures and silently ignored user configuration.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

diskann-benchmark/Cargo.toml:88

• The pipnn feature enables diskann-disk/pipnn but does not enable the optional diskann-disk dependency itself. Since the benchmark code behind cfg(feature = "pipnn") references diskann_disk::BuildAlgorithm (even outside the disk-index feature), building with --features pipnn can fail due to diskann-disk not being activated.

# Enable PiPNN graph construction.
pipnn = ["dep:diskann-pipnn", "diskann-disk/pipnn"]

diskann-benchmark/src/index/benchmarks.rs:241

• When build_algorithm selects PiPNN, this path bypasses run_build/single_or_multi_insert, so any multi_insert configuration in the input is silently ignored. That can mislead users who expect batched insertion settings to take effect; it should be rejected explicitly for PiPNN builds.

                    diskann_disk::BuildAlgorithm::PiPNN(parameters) => {
                        let data =
                            Arc::new(datafiles::load_dataset(datafiles::BinFile(build.data()))?);
                        build::pipnn_build(data, build, parameters)
                    }

• Files reviewed: 9/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

The PiPNN benchmark build path currently ignores multi_insert settings without rejecting them, which can lead to silently-misconfigured benchmark runs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

diskann-benchmark/src/index/build.rs:154

• When build_algorithm selects PiPNN, the multi_insert setting from IndexBuild is silently ignored (PiPNN always runs the one-shot batch build). This can lead to confusing configs where the user thinks multi-insert is active but it has no effect; it should be rejected explicitly for PiPNN builds.

    use anyhow::Context;

    let npoints = data.nrows();
    let dimensions = data.ncols();
    let metric = input.distance().into();
    let graph = input.try_as_config()?.build()?;

• Files reviewed: 9/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.