HelixWorld

English | 简体中文

HelixWorld is a real-time interactive audio-visual world model from Noiz AI.

Give it an image and a prompt. Walk forward or turn around — picture and sound update together. The spatial field turns with the camera. Audio is not a soundtrack laid on afterwards.

News

• 2026-10-12. We release HelixBench, our benchmark for interactive audio-visual world models, on Hugging Face. It evaluates whether synthesized sound fields track dynamic viewpoint motion.
• 2026-09-29. The HelixWorld technical report is now on arXiv.
• 2026-09-03. We release HelixWorld Preview v1 inference code and a preview checkpoint on Hugging Face. You can also roam an interactive world in the browser at helixworld.org.

Showcase

Continuous generation across 8 scenes. The keys on the left show movement; the keys on the right show camera control.

Savannah Ride	Crystal Cave
Sketch Studio	Vineyard at Dusk
Pencil Paris	Country Lane
Sunlit Street	Fountain Plaza

Highlights

1. Audio-Visual Synchronization

HelixWorld is a real-time interactive audio-visual world model in which visual scenes and camera-grounded spatial stereo sound co-evolve natively under user interaction. In a drumming example with a visible pause, HelixWorld's transients subside during the pause and resume with the motion, whereas the post-hoc dubbing baseline AudioX produces a transient during the pause.

2. Camera-Aligned Spatial Audio

Continuous 6-DoF camera poses are injected into the video branch via Projective Relative Positional Encoding (PRoPE), and cross-modal attention binds visual geometry with acoustic features, naturally transferring camera ego-motion into realistic stereo panning. On HelixBench, which evaluates whether synthesized sound fields track dynamic viewpoint motion, HelixWorld surpasses both EchoWM and cascaded video-to-audio (V2A) dubbing pipelines in spatial-acoustic consistency.

3. Real-Time Inference on a Single GPU

On a single NVIDIA H800 GPU, HelixWorld achieves a steady-state real-time factor (RTF) of 0.77 at 768×512 and 24 FPS, including video and audio decoding. On the WBench navigation split, it outperforms most open-source silent baselines in visual quality and control responsiveness.

4. User Study

In a blind pairwise user study with audio, HelixWorld is preferred over all baselines by a clear margin, winning the majority of votes in every matchup.

Under the same first frame and control inputs, HelixWorld retains scene structure during navigation and motion reversal:

Code

Preview v1

• Linux, Python 3.11, CUDA 12.x
• NVIDIA GPU, 80 GB VRAM recommended
• System ffmpeg and ffprobe

A reviewed BF16 single-GPU run uses about 70 GB of VRAM.

git clone https://github.com/NoizAI/HelixWorld.git
cd HelixWorld

conda create -n helixworld-preview python=3.11 -y
conda activate helixworld-preview
python -m pip install -r requirements.txt
python download_models.py

models/
├── text_encoder/gemma-3-12b/
└── weights/
    └── model.safetensors

CUDA_VISIBLE_DEVICES=0 ./run.sh \
  --image /path/to/first_frame.png \
  --prompt-file examples/prompt.json \
  --actions "W:5,right:5,stop:5" \
  --perspective first_person \
  --num-frames 121 \
  --output-dir outputs/demo

Edit examples/prompt.json, or pass --video-prompt, --audio-prompt, and --av-prompt. Output is a clean MP4 in <output-dir>/release/native/.

Actions: W A S D left right up down stop. Combine with + (W+D:8). The number after : is latent transitions; 121 frames have 15. The last segment may omit a duration.

Citation

@article{ke2026helixworld,
  title={HelixWorld: A Real-time Interactive Audio-Visual World Model},
  author={Ke, Lei and Pan, Jiahao and Tian, Zeyue and Wang, Jiaming and Huang, Haoyuan and Wu, Kam Man and Fang, Pengjun and Liu, Hongyu and Qi, Chenyang and Wang, Lin and others},
  journal={arXiv preprint arXiv:2609.38123},
  year={2026}
}

License

Code is Apache 2.0. The weight license is published with the Preview v1 checkpoint.

Community

Join our user groups to share results, ask questions, and get the latest updates.

WeChat

Discord