Repository navigation
feat(models): add LightlyViT model package - #1003
IgorSusmelj wants to merge 6 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ba4edd9925
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - [Object Detection](#object-detection) — fine-tune a LightlyViT-based LTDETR model; | ||
| supports loading custom pretrained backbone weights via `backbone_weights`. |
There was a problem hiding this comment.
The advertised object-detection flow cannot currently be used: a repo-wide search finds no LightlyViT entry in LTDETR_MODEL_REGISTRY, and LTDETRObjectDetection calls that registry before constructing a backbone and later restricts accepted backbones to ConvNeXt, DINOv2/DINOv3 ViTs, or ECViT. Consequently, users following this documentation cannot instantiate a LightlyViT LTDETR model or load the exported backbone_weights; add the required LTDETR configuration and wrapper support or remove this claim.
Useful? React with 👍 / 👎.
| if key_mask.dtype != torch.bool: | ||
| key_mask = key_mask > 0 |
There was a problem hiding this comment.
When a caller supplies a standard floating-point additive mask, allowed entries are 0 and excluded entries are negative (usually -inf), so key_mask > 0 marks every entry invalid. On the default linear-attention T/M/L variants, passing even an all-zero mask through execute_block therefore zeros every key and value and collapses the attention output instead of behaving like the unmasked call. Interpret additive masks consistently with GlobalSelfAttention, or explicitly reject unsupported floating masks.
Useful? React with 👍 / 👎.
| lightlyvit_l16_v1, | ||
| lightlyvit_m16_v1, | ||
| lightlyvit_s16_v1, |
There was a problem hiding this comment.
The package and documentation expose lightlyvit_t16_v1 as a supported model, but this public API imports and exports only the S/M/L factories. As a result, from lightly_train._models.lightlyvit import lightlyvit_t16_v1 fails while the equivalent imports for every other advertised variant work; include the Tiny factory in both the import list and __all__.
Useful? React with 👍 / 👎.
|
/review |
Why
LightlyViT is a compact ViT backbone family (T/S/M/L, patch size 16) that replaces most attention blocks with cheap local mixers (depthwise conv), keeping only a few global blocks. This gives strong quality per FLOP, which is interesting for robotics and embedded workloads with limited compute. Upstreaming the package makes LightlyViT checkpoints loadable and fine-tunable with LightlyTrain. Detection and segmentation adapters on top of it follow in separate PRs.
What changes
lightlyvitmodel package with variant factories fort16,s16,m16andl16, plus variant options such as the global mixer kind and the position encoding (learned or RoPE).forward_features, multi-scale features, activation checkpointing).model_args={"backbone_weights": ...}.How tested
tests/_models/lightlyvit).--fail-on-warningpasses.