Skip to content

Token length overflow and data filtering during Stage 2 training with provided scripts #3

Description

@bolo-web

I am trying to reproduce the ASID-Captioner-3B using the provided training scripts. Specifically, when running train_qwen2.5-omi-stage1-2.sh for Stage 2, I noticed that a significant number of samples from the single_attribute_0_30_s_youtube_v0_1.jsonl dataset are being filtered out or truncated because they exceed the defined max_length.

Details:
In train_qwen2.5-omi-stage1-2.sh, the configuration is:

VIDEO_TOTAL_PIXELS=20070400 (~20M pixels)
max_length 16384
Problem Analysis: Based on the Qwen2.5-Omni architecture:
The combination of visual tokens + structured text + system prompts often exceeds 16,384, causing the trainer to skip these valuable samples.
Interestingly, I noticed that in train_qwen2.5-omi-stage3.sh, the configuration is more relaxed regarding length but stricter on pixels:
VIDEO_TOTAL_PIXELS=12042240 (~12M pixels)
max_length 22384
Questions:
Is it intended for Stage 2 to have such a high pixel budget (20M) with a relatively small max_length (16384)?
Does this filtering significantly impact the model's ability to learn fine-grained attributes during the early stages?
For a successful reproduction, should I:
Increase max_length to 22384 for Stage 2?
Or decrease VIDEO_TOTAL_PIXELS to 12M (similar to Stage 3)?
Environment:
Model: Qwen2.5-Omni-3B
Framework: ms-swift v3.9.3
Script: train_qwen2.5-omi-stage1-2.sh

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions