BUZZY

Official implementation of BUZZY, a text-prior contrastive decoding method for scientific figure and multimodal multiple-choice QA.

[Paper] [Project page]

Method

Buzzy treats answer choices as an explicit text-induced prior and subtracts that prior during candidate scoring:

score(choice) = logit(choice | image, question) - alpha * logit(choice | question)

The default setting uses alpha = 0.5. The runner also includes the baselines used in our experiments: greedy decoding, prompt-enhanced decoding, VCD, ICD, and ALC/selection-bias debiasing.

Repository Contents

src/run_buzzy_eval.py          Main evaluation runner
scripts/run_buzzy_eval.sh      Minimal convenience wrapper

Large experiment outputs, datasets, local model checkpoints, paper files, one-off GPU scheduling scripts, and virtual environments are intentionally not included.

Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The runner expects an OpenAI-compatible vision-language model endpoint, such as vLLM or SGLang.

Data

The runner first checks data/ for evaluation datasets. If the separately provided review package is present at buzzy_review_datasets/data/, its saved MMBench and MMStar splits are detected automatically. Explicit CLI paths still take precedence.

Supported datasets:

• mac
• scifi
• mmsci
• mmbench
• mmstar
• vlmbias

Example layout:

data/
  MAC_Bench/
    test.jsonl
  images/
    MAC_Bench/
      ...
  scifi/
    test.parquet
  mmsci/
    test.json
    images/
      ...
  MMBench_validation/
    data-*.arrow
    dataset_info.json
    state.json
  MMStar_val/
    data-*.arrow
    dataset_info.json
    state.json

When the separately distributed review package is unpacked as buzzy_review_datasets/, MMBench and MMStar run without --input-mmbench or --input-mmstar:

python src/run_buzzy_eval.py --dataset mmbench --method buzzy
python src/run_buzzy_eval.py --dataset mmstar --method buzzy

Quick Start

Run Buzzy on MAC with an already running OpenAI-compatible VLM server:

python src/run_buzzy_eval.py \
  --dataset mac \
  --method buzzy \
  --contrastive-alpha 0.5 \
  --api-base http://127.0.0.1:30000/v1 \
  --output-jsonl results/mac_buzzy.jsonl

Run greedy or prompt-enhanced baselines:

python src/run_buzzy_eval.py \
  --dataset mac \
  --method greedy \
  --api-base http://127.0.0.1:30000/v1 \
  --output-jsonl results/mac_greedy.jsonl

python src/run_buzzy_eval.py \
  --dataset mac \
  --method prompt \
  --prompt-method visual_evidence \
  --api-base http://127.0.0.1:30000/v1 \
  --output-jsonl results/mac_prompt_enhanced.jsonl

Run contrastive baselines:

python src/run_buzzy_eval.py \
  --dataset mmbench \
  --method vcd_open \
  --api-base http://127.0.0.1:30000/v1 \
  --output-jsonl results/mmbench_vcd.jsonl

python src/run_buzzy_eval.py \
  --dataset mmbench \
  --method icd_open \
  --api-base http://127.0.0.1:30000/v1 \
  --output-jsonl results/mmbench_icd.jsonl

Methods

• greedy: candidate scoring with image and question.
• prompt: prompt-enhanced candidate scoring. Use with --prompt-method visual_evidence.
• buzzy: Buzzy with fixed text-prior subtraction.
• vcd_open: visual contrastive decoding baseline.
• icd_open: instruction contrastive decoding baseline.
• selection_bias: ALC-style selection-bias debiasing baseline.

Output

The runner writes one JSON object per evaluated example. Each record includes:

• gold and predicted answer
• correctness
• image-conditioned logits/probabilities
• text-only logits/probabilities when applicable
• contrastive details for Buzzy and contrastive baselines

The script prints accuracy and macro-F1 at the end of each run.