Question About Model Parameter Updates（self.training=false & requires_grad=true）

[D222097]

What will happen when self.vlm.training is set to False and self.vlm.parameters() has requires_grad set to True? Will the model parameters be updated if there is no dropout or batch normalization in the model?

Besides, I'm opening this issue due to an unexpected change in my model. The model includes self.dit, self.vlm, and other components. After completing self.train(), self.vlm.training is set to True, but when I enter training_step(), it switches to False. I'm not sure what is causing this behavior. Any suggestions regarding the potential reasons for this issue would be greatly appreciated.

def on_validation_end(self):
        self.train()
        # self.vlm.training is True
        # self.dit.training is True

def training_step(self, batch: Dict[str, Dict], batch_idx: int) -> torch.Tensor:
        # self.vlm.training is False
        # self.dit.training is True
        opt = self.optimizers()
        ...

[ska2704]

Yes, the parameters can still update with training=False. .eval() changes how modules use the training flag, but it doesn't disable autograd or freeze weights. Gradients must be enabled, the parameters must contribute to the loss and be included in the optimizer, and you still need backward and optimizer.step(). requires_grad=True by itself isn't enough. PyTorch explains the distinction here.

For the second part, check whether vlm was already in eval mode before validation. Lightning 2.6.0 saves each submodule's mode, then restores it after on_validation_end. If vlm was False and dit was True beforehand, self.train() makes both True inside your hook, but the later restoration puts vlm back to False.

A small CPU example on Lightning 2.6.0 reproduced that sequence. An eval-mode Linear also updated its weight from 1.0 to 0.8 after backward and an SGD step.

You can log the modes before Lightning switches to eval:

def on_validation_model_eval(self):
    print(self.vlm.training, self.dit.training)
    super().on_validation_model_eval()

If you want the whole model in training mode after validation, move that transition to:

def on_validation_model_train(self):
    super().on_validation_model_train()

The base hook calls self.trainer.model.train(), and this override passed the CPU check too. Only use it if every submodule should be in training mode. If the VLM is intentionally kept in eval mode, restoring that mode can be correct. Your Lightning version and the rest of the model/callback code would help confirm whether this explains your case.

Prepared with Codex; the examples were checked on CPU.