Whisper Support for model builder

[Siddharth Kolipara (sidk1023)]

I see that whisper is added to the example scripts but is there any plan to add whisper model building into the builder.py script? If yes, what are the timelines for the same?

[kunal-vaishnavi]

For now, you can get everything you need to run Whisper with ONNX Runtime GenAI by following the Whisper-specific README here. The steps are also mentioned here.

[Henrik Levring (hlevring)]

I am trying to follow the readme in the above comment, but it seems focused on cuda exclusively.

I am looking at this section
Exporting + Optimizing + Quantizing Whisper with Beam Search

My objective is to export openai/whisper-large-v3-turbo to onnx fp16 format that will work well with ort-genai DirectML EP (and potentially also with OpenVino GPU EP and TensorRT-RTX EP). But at minimum at least with DirectML EP.

[kunal-vaishnavi]

The original approach to have the BeamSearch op in the ONNX model was supported for the CPU EP, CUDA EP, and ROCm EP. The ROCm EP was subsequently deprecated and now the original approach is supported for the CPU EP and CUDA EP.

The new approach to use ONNX Runtime GenAI is currently supported for the CPU EP and CUDA EP. To use the DirectML EP, you can try producing the ONNX models with the new approach while still specifying the CUDA EP as the target. If you follow the README examples, you will need to omit --output_cross_qk as it is not supported in other EPs besides CPU and CUDA.

For these unsupported EPs, only use the command-line arguments in the "Conversion Process Args" group and do not use --use_decoder_masked_mha. You can see the grouped command-line arguments when running with --help.

[kunal-vaishnavi]

Whisper has now been added into the model builder.