Repository navigation

Pipeline parallelism continued - #399

Merged

BlueCrescent merged 116 commits into

mainfrom

pipeline_parallelism_fix

Nov 7, 2025

Merged

Pipeline parallelism continued#399
BlueCrescent merged 116 commits into
mainfrom
pipeline_parallelism_fix

Commits on Aug 6, 2025
- feat: implemented stage FQN generation for pipeline parallelism
  le1nux
  committed
  1e80bbb
Commits on Aug 7, 2025
- feat: added FQNs per stage calculation
  le1nux
  committed
  ed93d28
Commits on Aug 15, 2025
- feat: generic FQN-based PP staging
  le1nux
  committed
  6241ea8
- feat: added PP configs
  le1nux
  committed
  0ba8fbc
- feat: wired up PP within dependency graph
  le1nux
  committed
  4a41b6c
- feat: added FQN stages generator
  le1nux
  committed
  ee529b7
Commits on Aug 18, 2025
- feat: implemented scheduled pipeline
  le1nux
  committed
  625de59
- feat: wired up scheduled and staged pipelines.
  le1nux
  committed
  9677bd6
- feat: added PP test config
  le1nux
  committed
  7ac9edf
Commits on Aug 19, 2025
- refactor: staging is now fully instantiable
  le1nux
  committed
  d9f63c1
- feat: drafted pp e2e test for fwd/bwd pass
  le1nux
  committed
  83c87b9
Commits on Aug 29, 2025
- refactor: renamings in the context of PP
  le1nux
  committed
  95f2470
- chore: drafted the first PP test.
  le1nux
  committed
  521e586
- chore: pp config fixes
  le1nux
  committed
  002b0ae
Commits on Sep 5, 2025
- feat: Make test for pipeline parallelism work
  rrutmann
  committed
  1d4943f
Commits on Sep 8, 2025
Commits on Sep 9, 2025
- feat: Use scheduled_pipeline for forwad backward pass
  rrutmann
  committed
  27ad56d
- feat: Use scheduled_pipeline for training
  rrutmann
  committed
  41c4f36
- feat: Use scheduled_pipe in evaluation
  rrutmann
  committed
  6f3d5da
- test: Print losses if test fails
  rrutmann
  committed
  9b85334
- chore: Run evaluation before training
  rrutmann
  committed
  84e2702
- chore: Increase microbatch size
  rrutmann
  committed
  32fbe94
Commits on Sep 10, 2025
- fix: Use dp size instead of world size for last batch aggregation
  rrutmann
  committed
  61ab311
- docs: Add TODOs for later check
  rrutmann
  committed
  6952bcc
- fix: Train before evaluation so that pp is initialized for backwards
  rrutmann
  committed
  90dbe51
Commits on Sep 12, 2025
- fix: Add missing parameter seed to GPT2LLMConfig
  rrutmann
  committed
  49df7d6
Commits on Sep 15, 2025
- fix: Retrieve all PP ranks for gradient clipping
  rrutmann
  committed
  7996a29
- test: Add new parameter num_data_parallel_ranks to Trainer
  rrutmann
  committed
  cbddcbc
- fix: Make FSDP1GradientClipperConfig independent of device_mesh
  rrutmann
  committed
  56a917a
- fix: Handle optional device_mesh correctly
  rrutmann
  committed
  eb47aa9
Commits on Sep 17, 2025
- feat: Consider pipeline parallelism in tensor pallelization
  rrutmann
  committed
  d228351
- test: Use the same data on each rank & test tensor parallelism
  rrutmann
  committed
  55dad72
- refactor(parallelism): Some clean-up.
  BlueCrescent
  committed
  b6a1e2d
Commits on Sep 18, 2025
- chore: Merge branch 'pipeline_parallelism_fix' of github.com:Modalities/modalities into pipeline_parallelism_fix
  rrutmann
  committed
  16a51af
Commits on Sep 19, 2025
- test: Update configs for parallelization testing
  rrutmann
  committed
  c49895a
- test: Use correct length to create test sequences
  rrutmann
  committed
  f685fc5
- test: Use realistic std for model initialization
  rrutmann
  committed
  c07fcf6
- fix: Remove unused third dimension for reduced_losses
  rrutmann
  committed
  5019bbb
- refactor: Remove unused filtering
  rrutmann
  committed
  a08e555
Commits on Sep 22, 2025
- fix: Aggregate loss of last train batch correct across pp ranks
  rrutmann
  committed
  45b5418
- docs: Add example config for pipeline and tensor parallelism
  rrutmann
  committed
  a394ab0
- docs: Add docstrings and type hints
  rrutmann
  committed
  cae050e
- docs: Add type hints and docstrings
  rrutmann
  committed
  6952230
- fix: Check if parallelism method is initialized
  rrutmann
  committed
  ffa032c
- docs: Add new parameter in docstring
  rrutmann
  committed
  8d418a1
Commits on Sep 23, 2025
- test: Run only one PP only test
  rrutmann
  committed
  fffd0a1
Commits on Sep 24, 2025
- refactor: Addressed copilot review
  rrutmann
  committed
  049472f
Commits on Oct 15, 2025
- chore: Remove requirements for python and torch
  rrutmann
  committed
  608c7fc
Commits on Oct 17, 2025
Commits on Oct 20, 2025
- test(checkpointing): Made dcp checkpointing test terminate correctly when failing in one of multiple subprocesses.
  BlueCrescent
  committed
  cd9f595
Commits on Oct 21, 2025
Commits on Oct 23, 2025
Commits on Oct 24, 2025
Commits on Oct 27, 2025
Commits on Nov 3, 2025
- Merge: main into current branch
  rrutmann
  committed
  5013420
- fix: Fix spelling mistake
  rrutmann
  committed
  bf778c1
- Fix: Adapt configts to latest changes
  rrutmann
  committed
  79f3a6d
- Fix: Adapt configs to latest configs
  rrutmann
  committed
  4aa144b
- fix: Compute peak memory correctly on cpu device
  rrutmann
  committed
  af68c39
- test: Require A100 GPUs for mfu test
  rrutmann
  committed
  e4d29ca
- test: Adapt tests to latest changes
  rrutmann
  committed
  e8781a3
- test(parallelism): Adapted fsdp2+tp+tt test to recent changes.
  
  BlueCrescent
  authored and
  rrutmann
  committed
  8d53b3c
- Update tests/fsdp2_parallelization/pipeline_parallelism/test_pp_fwd_bwd_pass.py
  
  rrutmann
  and
  Copilot
  committed
  dfb8ce9
- Update src/modalities/models/parallelism/pipeline_parallelism.py
  
  rrutmann
  and
  Copilot
  committed
  37f3822
Commits on Nov 4, 2025
- test(end2end): Reworked warmstart tests to not require torchrun and allow different world sizes for second run.
  BlueCrescent
  committed
  4f3055f
- refactor: Implemented some PR comments.s
  BlueCrescent
  committed
  f5fc017
- fix: Make device_mesh required for fsdp2
  rrutmann
  committed
  8d0c012
- fix: Adapt config to latest changes
  rrutmann
  committed
  e25ba8e
- docs: Add docstring for seed
  rrutmann
  committed
  36525ce
- docs: Fix comment regarding loss reduction
  rrutmann
  committed
  6d9886c
- fix: Fix typo
  rrutmann
  committed
  70a6370
- docs: Add comment for clarity on optimizer groups
  rrutmann
  committed
  48203b7
- docs: Add missing return types
  rrutmann
  committed
  7d95fac
- fix: Compute global_rank correctly with other_degree
  rrutmann
  committed
  868f86e
Commits on Nov 5, 2025
- refactor: Put tests and utilities in correct files
  rrutmann
  committed
  1adb4e3
- refactor: Use new function has_parallelism_method
  rrutmann
  committed
  ab12f5a
- test: Add test for pipeline and tensor parallelism
  rrutmann
  committed
  3bedd80
- test: Increase size of TestDataset
  rrutmann
  committed
  7b4139d
- test(parallelism): Refactored PP unit test.
  BlueCrescent
  committed
  b558ef8
- chore: Removed commented out section in test config.
  BlueCrescent
  committed
  238e72a
- refactor: Additional assert for PR.
  BlueCrescent
  committed
  c6cbdeb
- test: Check for exact samples on each rank
  rrutmann
  committed
  c9ce2e1
- docs(parallelism): Better code comment.
  BlueCrescent
  committed
  9bc28c2
- test(multi_processing): Moved duplicate monitor_child_processes() helpers to single implementation in utility.
  BlueCrescent
  committed
  e446fa3
Commits on Nov 6, 2025
Commits on Nov 7, 2025
- revert(evaluation): Reactivated evaluation before training.
  BlueCrescent
  committed
  36c8bcd
- refactor: Define function to get parallel rank in device_mesh.py
  rrutmann
  committed
  5700a94
- test: Use the same test data only on data parallel ranks
  rrutmann
  committed
  0fce000
- feat: Log rank information for each rank
  rrutmann
  committed
  b37cdeb
- refactor: Minor cleanup.
  BlueCrescent
  committed
  01c7b05
- refactor: Replaced asserts with exceptions.
  BlueCrescent
  committed
  91e32cc
- test(data): Fixed device mesh mocks for current requirements.
  BlueCrescent
  committed
  a3e2a81
- test(data): Added additional check in dataloader test.
  BlueCrescent
  committed
  51b7fa2
- fix(optimizer): Switcheroo in weight decay groups.
  BlueCrescent
  committed
  f98ea35
- chore: Merge remote-tracking branch 'origin/main' into pipeline_parallelism_fix
  BlueCrescent
  committed
  2a90c0e

Commits

Commits on Aug 6, 2025

Commits on Aug 7, 2025

Commits on Aug 15, 2025

Commits on Aug 18, 2025

Commits on Aug 19, 2025

Commits on Aug 29, 2025

Commits on Sep 5, 2025

Commits on Sep 8, 2025

Commits on Sep 9, 2025

Commits on Sep 10, 2025

Commits on Sep 12, 2025

Commits on Sep 15, 2025

Commits on Sep 17, 2025

Commits on Sep 18, 2025

Commits on Sep 19, 2025

Commits on Sep 22, 2025

Commits on Sep 23, 2025

Commits on Sep 24, 2025

Commits on Oct 15, 2025

Commits on Oct 17, 2025

Commits on Oct 20, 2025

Commits on Oct 21, 2025

Commits on Oct 23, 2025

Commits on Oct 24, 2025

Commits on Oct 27, 2025

Commits on Nov 3, 2025

Commits on Nov 4, 2025

Commits on Nov 5, 2025

Commits on Nov 6, 2025

Commits on Nov 7, 2025