Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Style Shift Generalization: ERM and MixStyle on PACS

Homework 1 for DATA 405 Trustworthy AI (Prof. Jindong Wang, Fall 2026). The goal is to show a model that does well on familiar data and worse when the data changes, explain why, and test one method that helps it generalize.

Status: all 24 runs done (ERM and MixStyle, 4 held out domains, 3 seeds); results and figures are in results/. The 2 page report is written in LaTeX in report/.

The assignment

Due September 25, 2026 at midnight.

Build one distribution shift experiment with at least two test conditions: one close to the training distribution and one with a meaningful shift. Any model, dataset, or open source code is allowed, as long as I design the whole pipeline.

The professor's spec from Lecture 03:

  • Use a dataset with distribution shift (PACS, Office-Home, or CIFAR-10-C).
  • Train a standard model (ResNet-18 or ViT-small) with ERM.
  • Evaluate on an in distribution (ID) test set and an out of distribution (OOD) test set.
  • Optionally implement one generalization method, such as data augmentation or a domain generalization baseline like CORAL or MixStyle.
  • Write a 2 page report.

Minimum for a B: complete code for dataset loading, model training, and ID and OOD evaluation, plus a clear description of the whole experiment.

Bonus: a personal reflection on the experiment, and an implemented generalization method with results.

The report has to cover:

  1. The model, the data, and the distribution shift.
  2. Performance before and after the shift.
  3. At least one table or figure showing the difference.
  4. Why the performance changed.
  5. One reasonable way the system could generalize better.

The experiment

Data: PACS

PACS has 7 object classes (dog, elephant, giraffe, guitar, horse, house, person) drawn in 4 visual domains: Photo, Art painting, Cartoon, and Sketch. The shift is style: the same dog looks very different as a photo and as a pencil sketch.

Setup is leave one domain out. Train on three domains and test on the fourth, which the model never sees during training. Every domain is held out in turn. Sketch is the headline target because it is the furthest from the others.

Each domain is split 70/10/20 into train, val, and test, stratified by class, and saved in splits/pacs.csv.

  • Train: the train rows of the three source domains.
  • Validation: their val rows, used only for model selection.
  • ID test: their test rows.
  • OOD test: every image of the held out domain.

Model

ResNet-18 pretrained on ImageNet, fine tuned on PACS.

Methods

Run What it is Role
ERM Plain training on the pooled source domains Baseline, required
MixStyle Mixes per channel feature statistics (style) between images after ResNet layers 1 to 3, during training only Implemented, the main method
DANN A domain discriminator plus a gradient reversal layer, so the features stop revealing which domain an image came from Not implemented, to leave time for the analysis and report

Why MixStyle is the lead: it is about 15 lines with no extra loss or network, it trains stably, it needs no domain labels, and the professor named it in the spec. It fits PACS well because the domains differ mostly in style. Because it can only create styles between the training styles, it was expected to help least on Sketch. Sketch gained the most instead (see Results).

Evaluation protocol

  • Choose hyperparameters and the stopping epoch using only the source domains' validation split. Never tune on the held out domain.
  • Run each configuration with 3 seeds and report the mean and spread.
  • Metrics: ID accuracy, OOD accuracy, the gap between them, and per domain accuracy.

Analysis beyond accuracy

  • Domain probe: freeze the trained network, summarize each channel after every ResNet stage by its mean and std (the style MixStyle mixes), train a linear classifier to tell source images from held out ones, and convert its error into the A distance, $d_A = 2(1 - 2\epsilon)$. This shows at which layer each method keeps or removes domain information, compared with the ImageNet weights before fine tuning.
  • t-SNE of the features, colored by domain and by class, for ERM and each method.

Planned figures and tables

  • ID vs OOD accuracy per method and held out domain (table, and a dot plot whose line lengths are the gap).
  • Learning curves with Sketch held out, with the seed spread shaded.
  • Confusion matrices on held out Sketch, ERM next to MixStyle.
  • Domain probe $d_A$ per layer and method (notebook only, left out of the report for space).
  • t-SNE plots colored by domain and by class (notebook only).

Results

Accuracy in percent, mean ± standard deviation over 3 seeds, at the epoch with the best source validation accuracy. ID is the source domains' test split; OOD is the held out domain.

Held out ERM ID ERM OOD ERM gap MixStyle ID MixStyle OOD MixStyle gap
Art 95.2 ± 0.6 76.3 ± 1.8 18.9 ± 1.4 95.4 ± 0.1 81.3 ± 1.1 14.1 ± 1.1
Cartoon 94.3 ± 0.2 73.0 ± 1.9 21.3 ± 1.7 95.0 ± 0.3 77.4 ± 0.8 17.6 ± 1.1
Photo 93.9 ± 0.2 95.4 ± 0.6 -1.5 ± 0.5 93.9 ± 0.3 95.5 ± 0.3 -1.7 ± 0.6
Sketch 95.3 ± 0.3 64.8 ± 0.6 30.5 ± 0.8 95.5 ± 0.3 72.6 ± 1.8 22.9 ± 2.1
  • ID accuracy is about 95% for both methods; the shift costs ERM up to 30 points on Sketch.
  • MixStyle raises OOD accuracy by 4 to 8 points on Art, Cartoon, and Sketch without changing ID accuracy.
  • Sketch gains the most (+7.8), the opposite of the plan's expectation that MixStyle would help least there.
  • Holding out Photo is barely a shift, because the ImageNet pretraining already covers photos.
  • The domain probe shows MixStyle does not make the domains look alike: its A distance is within about 0.05 of ERM's at every layer, and Sketch stays fully separable. The gain comes from the classifier tolerating style changes, not from removing domain information.
  • These numbers are within about 3 points of the ResNet-18 results in the MixStyle paper.
  • On held out Sketch, ERM predicts "elephant" for 46% of people, 28% of dogs, and 18% of houses (pooled over seeds). MixStyle cuts person to elephant to 20% and house to elephant to 7%. One likely reason: elephant is the gray, low texture class in the source domains, so ERM uses color and texture as a shortcut that fails on sketches. This has not been checked against the source images' color statistics.

ID and OOD accuracy per held out domain

Confusion matrices on held out Sketch, ERM and MixStyle

Figures are in results/figures/, built by notebooks/analysis.ipynb.

How it works

The whole pipeline, from download to report:

flowchart TD
    A["scripts/download_pacs.py<br/>data/pacs/domain/class/*.jpg"] --> B["scripts/make_splits.py<br/>splits/pacs.csv: train / val / test per image"]
    B --> C["styleshift sweep<br/>2 methods x 4 targets x 3 seeds"]
    C -->|"calls run(cfg) 24 times,<br/>skips runs with metrics.json"| D

    subgraph D["styleshift train: train.py run()"]
        direction TB
        D1["config.py: TrainConfig<br/>method, target, seed, lr ..."] --> D2["data.py: split_rows<br/>sources = 3 domains, target = held out"]
        D2 --> D3["build_loaders<br/>train | val | id_test | ood_test"]
        D3 --> D4["methods.py: build_algorithm<br/>ERM or ERM + MixStyle"]
        D4 --> D5["train_epoch<br/>one pass over source train"]
        D5 --> D6["evaluate val, ID test, OOD test"]
        D6 --> D7{"best val<br/>so far?"}
        D7 -->|yes| D8["keep weights and this epoch's numbers"]
        D7 -->|no| D9
        D8 --> D9{"epoch < 50?"}
        D9 -->|yes| D5
        D9 -->|no| D10["write history.csv, model.pt, metrics.json<br/>runs/method/target/seedK/"]
    end

    D --> E["styleshift collect<br/>results/runs.csv, summary.csv, history.csv"]
    D --> F["styleshift probe<br/>per layer domain probe + t-SNE<br/>results/probe.csv, results/tsne/"]
    D --> K["styleshift confusion<br/>true vs predicted class counts<br/>results/confusion.csv"]
    E --> G["notebooks/analysis.ipynb<br/>results/figures/*.png and *.pdf<br/>report/table_id_ood.tex"]
    F --> G
    K --> G
    G --> H["report/main.tex<br/>2 page report"]
Loading

One training step with MixStyle. The ERM baseline is the same network without the three MixStyle boxes, which is the only difference between the two methods.

flowchart LR
    X["batch of 64 source images<br/>x, class y, domain d"] --> S["stem"]
    S --> L1["layer1"] --> M1["MixStyle"] --> L2["layer2"] --> M2["MixStyle"] --> L3["layer3"] --> M3["MixStyle"] --> L4["layer4"]
    L4 --> P["avg pool<br/>512 features"] --> C["classifier<br/>7 classes"]
    C --> LOSS["cross entropy vs y"] --> BW["backward + SGD step"]

    subgraph MS["each MixStyle, training only, 50% of batches"]
        direction TB
        a["mu, sig per image per channel<br/>= the style"] --> b["normalize: (x - mu) / sig<br/>= content without style"]
        b --> c["pick a random partner image<br/>blend: lmda * own + (1 - lmda) * partner"]
        c --> e["x_normed * sig_mix + mu_mix<br/>= same content, mixed style"]
    end
Loading

Layout

style_shift_generalization/
  README.md
  pyproject.toml
  scripts/
    download_pacs.py     fetches PACS into data/pacs/<domain>/<class>/
    make_splits.py       writes splits/pacs.csv (already committed)
  splits/
    pacs.csv             train, val, or test for every image
  src/style_shift_generalization/
    __init__.py          the styleshift command: train, sweep, collect, probe, confusion
    config.py            TrainConfig, every setting with its default
    data.py              PACS dataset, transforms, leave one domain out loaders
    mixstyle.py          MixStyle module
    models.py            ResNet-18 with optional MixStyle after layers 1 to 3
    methods.py           ERM and MixStyle behind one Algorithm interface
    train.py             one run: train, evaluate each epoch, save the best val epoch
    sweep.py             every method x held out domain x seed, resumable
    collect.py           runs/ -> results/runs.csv, summary.csv, history.csv
    probe.py             domain probe (A distance) and t-SNE
    confusion.py         true vs predicted class counts on ID and OOD test
  tests/                 MixStyle behavior and split leakage checks
  notebooks/
    analysis.ipynb       reads results/, draws the report figures
  runs/                  one folder per run with its checkpoint (not committed)
  results/               metrics CSVs and figures (committed)
  report/
    main.tex             the 2 page write up
    refs.bib             references
    table_id_ood.tex     results table, written by the notebook
    Makefile             builds main.pdf

Setup

The project uses uv and Python 3.13.

uv sync                # creates .venv and installs everything from uv.lock
source .venv/bin/activate      # or .venv/bin/activate.fish in fish

Download PACS (about 190 MB) and export it to data/pacs/:

uv run python scripts/download_pacs.py

The split in splits/pacs.csv is committed, so there is no need to regenerate it. scripts/make_splits.py rebuilds it identically, and only overwrites it with --force.

Check that PyTorch sees the GPU:

uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

PyTorch comes from the official CUDA 13.0 wheel index, set in pyproject.toml under [tool.uv.sources]. The RTX 5060 is a Blackwell card (compute capability 12.0), which needs a CUDA 12.8 or newer build.

Packages

Package Used for
torch Models, training, GPU
torchvision ResNet-18 with ImageNet weights, image transforms, ImageFolder loading
numpy Array math, the MixStyle and probe calculations
pandas Collecting results across runs and seeds into tables
scikit-learn Linear domain probe, t-SNE, metrics
matplotlib Figures for the report
pillow Reading the PACS image files
tqdm Progress bars during training and export
huggingface_hub Downloading the pinned PACS parquet file
pyarrow Reading the parquet file

Dev only (uv sync installs these too):

Package Used for
ruff Lint and format
pytest Tests, for example that MixStyle leaves $\mu$ and $\sigma$ as expected
ipykernel Running exploration notebooks against this venv

Running the experiments

Each run trains one method with one held out domain and one seed:

uv run styleshift train --method erm --target sketch --seed 0
uv run styleshift train --method mixstyle --target sketch --seed 0

Every setting in src/style_shift_generalization/config.py is also a flag, for example --epochs 1 for a quick check. uv run styleshift train --help lists them all. A run writes config.json, history.csv, model.pt, and metrics.json to runs/<method>/<target>/seed<k>/.

The full experiment is 2 methods x 4 held out domains x 3 seeds, 24 runs of about 8 minutes each on an RTX 5060, about 3 hours 20 minutes in total:

uv run styleshift sweep       # skips any run whose metrics.json already exists, so it resumes after a stop
uv run styleshift collect     # writes results/runs.csv, summary.csv, history.csv
uv run styleshift probe       # writes results/probe.csv and results/tsne/
uv run styleshift confusion   # writes results/confusion.csv

uv run styleshift sweep --methods mixstyle --targets sketch --seeds 0 1 2 runs a subset. Then open notebooks/analysis.ipynb and run all cells to build the table and figures in results/figures/ and the LaTeX table in report/.

Building the report

The report is a two column LaTeX paper, at most 2 pages before the references, and needs pdflatex and bibtex (TeX Live). The built PDF is not committed; the submitted PDF links back to this repo.

cd report && make      # writes report/main.pdf
make clean             # removes build files and the PDF

Red TODO text marks sections still to be written. Numbers are never typed into main.tex: the table comes from the notebook and the figures from results/figures/*.pdf, so rerunning the notebook updates the report.

Tests and lint

uv run pytest
uv run ruff check . && uv run ruff format --check .

Environment

  • GPU: NVIDIA RTX 5060 (8 GB), driver supporting CUDA 13.4.
  • Verified: torch 2.14.0+cu130 detects the GPU and runs a ResNet-18 forward pass.
  • Each run took about 8 minutes, and the full 24 run sweep about 3 hours 20 minutes.

Keeping the public repo clean

This repo will be public and linked in the submission.

  • Do not commit the PACS data. data/ is in .gitignore, and the download script recreates it.
  • Do not commit model checkpoints.
  • Do not commit the course slides or screenshots of them. Link to the course instead.
  • Read any API keys or tokens (for example, experiment tracking) from environment variables. Never commit them.
  • Keep the license notices of any adapted code.

References

Code this project draws on:

  • DeepDG, the course's recommended domain generalization toolkit, which includes ERM, DANN, CORAL, and loaders for PACS.
  • MixStyle official implementation.
  • DomainBed, which includes a PACS download script and the standard evaluation protocol.

Papers:

  • Zhou et al., Domain Generalization with MixStyle, ICLR 2021. arXiv 2104.02008
  • Ganin et al., Domain-Adversarial Training of Neural Networks, JMLR 2016. arXiv 1505.07818
  • Li et al., Deeper, Broader and Artier Domain Generalization (the PACS dataset), ICCV 2017. arXiv 1710.03077
  • Gulrajani and Lopez-Paz, In Search of Lost Domain Generalization, ICLR 2021. arXiv 2007.01434
  • Wang et al., Generalizing to Unseen Domains: A Survey on Domain Generalization, TKDE 2022. arXiv 2103.03097

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages