ARIA-Rubrics: Audio Reasoning Integrity Assessment for Audio Large Language Models

Official codebase for evaluating acoustic perception and Chain-of-Thought (CoT) reasoning quality in Audio Large Language Models (ALLMs).

The benchmark contains two evaluation tracks:

• MMAR — Multi-Modal Audio Reasoning
• MMAU — Multi-Modal Audio Understanding

---

📌 ARIA Framework Overview

---

📁 Repository Structure

.
├── generate/                 # CoT generation pipelines
│   ├── generate_cot_audio_reasoner.py
│   ├── generate_cot_audsemthinker.py
│   ├── generate_cot_gemini.py
│   ├── generate_cot_gemma3n.py
│   ├── generate_cot_gpt4o.py
│   ├── generate_cot_qwen2audio.py
│   ├── generate_cot_qwen25omni.py
│   ├── generate_cot_r1aqa.py
│   └── generate_cot_phi4.py
│
├── MMAR/                     # Multi-Modal Audio Reasoning
│   ├── audio/
│   ├── m1.py
│   ├── m2.py
│   ├── m3.py
│   ├── m4.py
│   ├── m5.py
│   └── m6.py
│
├── MMAU/                     # Multi-Modal Audio Understanding
│   ├── audio/
│   ├── m1.py
│   ├── m2.py
│   ├── m3.py
│   ├── m4.py
│   ├── m5.py
│   └── m6.py
│
├── ontology.json
├── words.txt
├── ARIA_Rubrics_Pipeline.pdf
├── requirements.txt
└── README.md

---

📊 Metrics

• M1 — Acoustic Grounding Score
• M2 — Inter-step Coherence Score
• M3 — Reasoning Chain Coverage
• M4 — Reasoning Step Causality
• M5 — Reasoning Progress Score
• M6 — Audio Lexical Density

---

📥 Download Datasets

MMAR

huggingface-cli download BoJack/MMAR \
    --repo-type dataset \
    --local-dir ./MMAR

MMAU-mini

huggingface-cli download AudioLLMs/MMAU-mini \
    --repo-type dataset \
    --local-dir ./MMAU

---

🚀 Usage Example

Install dependencies:

pip install -r requirements.txt

Run CoT generation pipelines:

cd generate
python generate_cot_audio_reasoner.py

---

Run evaluation scripts:

cd MMAR
# cd MMAU
python m1.py

📚 Resources

• AudioSet Ontology used for M6 Metric: https://research.google.com/audioset/download.html
• ontology.json — AudioSet ontology metadata
• words.txt — Curated acoustic vocabulary for lexical evaluation