Add Echo modular pipeline - #14696
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
yiyixuxu
left a comment
There was a problem hiding this comment.
thanks, I left some feedbacks
| @@ -0,0 +1,449 @@ | |||
| # Copyright 2026 The HuggingFace Team. All rights reserved. | |||
There was a problem hiding this comment.
can you change file names to remove the echo_ prefix?
|
|
||
|
|
||
| def _get_prompt_embeds( | ||
| components, |
There was a problem hiding this comment.
| components, | |
| tokenizer, | |
| text_encoder, |
for module-level functions, let's take explicit argument instead of componennts
| OutputParam( | ||
| "batch_size", | ||
| type_hint=int, | ||
| description="Number of prompts before per-prompt expansion.", | ||
| ), | ||
| OutputParam("dtype", type_hint=torch.dtype, description="Prompt embedding dtype."), |
There was a problem hiding this comment.
| OutputParam( | |
| "batch_size", | |
| type_hint=int, | |
| description="Number of prompts before per-prompt expansion.", | |
| ), | |
| OutputParam("dtype", type_hint=torch.dtype, description="Prompt embedding dtype."), |
so text_encoder step can be used on its own to compute prompt_embeds -> we want to minimize outputs, batch_size and dtype can easily derived later
|
|
||
|
|
||
| # auto_docstring | ||
| class EchoTextConditioningStep(SequentialPipelineBlocks): |
There was a problem hiding this comment.
let's move all the assembled blocks into modular_blocks_echo.py
| return (latents - latents_mean) / latents_std | ||
|
|
||
|
|
||
| def _as_list(value): |
There was a problem hiding this comment.
can we in-line this function?
|
|
||
| @staticmethod | ||
| def _audio_memory_coords( | ||
| components, |
There was a problem hiding this comment.
| components, | |
| transformer, |
let's make methods/functions argument more explicit
|
|
||
| @staticmethod | ||
| def _video_memory_coords( | ||
| components, |
There was a problem hiding this comment.
| components, | |
| transformer, |
| return torch.cat(coords, dim=2) | ||
|
|
||
| @torch.no_grad() | ||
| def __call__(self, components, state: PipelineState) -> PipelineState: |
There was a problem hiding this comment.
can we keep only the vae encoding in this step, and move the part that requires transformer into a block in before_denoise.py? so that the encoder step do not have to declare transformer as a component
| def expected_components(self) -> list[ComponentSpec]: | ||
| return [ | ||
| ComponentSpec("transformer", LTX2VideoTransformer3DModel), | ||
| ComponentSpec("audio_vae", AutoencoderKLLTX2Audio), |
There was a problem hiding this comment.
I think it's not needed, the code did not use it - also, vaes should only be required in enocder/decoder steps
| ComponentSpec("audio_vae", AutoencoderKLLTX2Audio), |
| """ | ||
|
|
||
| model_name = "echo" | ||
| block_classes = [ |
There was a problem hiding this comment.
i think we can have a
text_encoder, "vae_encoder","denoise","decode"
see some write up on assembling the blocksets https://github.com/huggingface/diffusers/blob/main/.ai/references/modular.md#key-pattern-flat-blocksets
What does this PR do?
Adds official Diffusers Modular Pipeline support for Echo, a long-video model that generates synchronized video and audio with cross-shot memory.
This supersedes #13910 with a modular implementation that incorporates the earlier review feedback:
diffusers.modular_pipelines.echopackage and the publicEchoModularPipeline/EchoBlocksAPIsscripts/instead of adding an original-checkpoint runtime pipelineOfficial resources:
Closes #13909
Supersedes #13910
Testing
make stylemake fix-copiesmake qualityPYTHONPATH=src python -m pytest -q tests/modular_pipelines/echo/test_modular_pipeline_echo.pyAI-assisted self-review
Codex was used to help review, refactor, test, and prepare this contribution. The final diff was reviewed using the repository's model-integration and self-review guidance.
The final self-review found no blocking correctness issues. Echo has its own public package and naming throughout; existing lower-level Diffusers component types are reused only as implementation dependencies. Memory-slot count, short-audio handling, variable audio lengths, response-aware 9.62-second cropping, deterministic DMD re-noising, mixed-precision decoding, loading, save/load, workflow, and memory behavior are covered by the test suite.
Validation scope: the final pass used the public
Echo-Team/tiny-echo-modular-pipefixture on CPU. A full production-checkpoint conversion and end-to-end GPU inference run was not repeated in this final pass.Before submitting
self-reviewskill on the diff?Who can review?
@asomoza @yiyixuxu