Homework 1 for DATA 405 Trustworthy AI (Prof. Jindong Wang, Fall 2026). The goal is to show a model that does well on familiar data and worse when the data changes, explain why, and test one method that helps it generalize.
Status: all 24 runs done (ERM and MixStyle, 4 held out domains, 3 seeds); results and figures are in results/.
The 2 page report is written in LaTeX in report/.
Due September 25, 2026 at midnight.
Build one distribution shift experiment with at least two test conditions: one close to the training distribution and one with a meaningful shift. Any model, dataset, or open source code is allowed, as long as I design the whole pipeline.
The professor's spec from Lecture 03:
- Use a dataset with distribution shift (PACS, Office-Home, or CIFAR-10-C).
- Train a standard model (ResNet-18 or ViT-small) with ERM.
- Evaluate on an in distribution (ID) test set and an out of distribution (OOD) test set.
- Optionally implement one generalization method, such as data augmentation or a domain generalization baseline like CORAL or MixStyle.
- Write a 2 page report.
Minimum for a B: complete code for dataset loading, model training, and ID and OOD evaluation, plus a clear description of the whole experiment.
Bonus: a personal reflection on the experiment, and an implemented generalization method with results.
The report has to cover:
- The model, the data, and the distribution shift.
- Performance before and after the shift.
- At least one table or figure showing the difference.
- Why the performance changed.
- One reasonable way the system could generalize better.
PACS has 7 object classes (dog, elephant, giraffe, guitar, horse, house, person) drawn in 4 visual domains: Photo, Art painting, Cartoon, and Sketch. The shift is style: the same dog looks very different as a photo and as a pencil sketch.
Setup is leave one domain out. Train on three domains and test on the fourth, which the model never sees during training. Every domain is held out in turn. Sketch is the headline target because it is the furthest from the others.
Each domain is split 70/10/20 into train, val, and test, stratified by class, and saved in splits/pacs.csv.
- Train: the train rows of the three source domains.
- Validation: their val rows, used only for model selection.
- ID test: their test rows.
- OOD test: every image of the held out domain.
ResNet-18 pretrained on ImageNet, fine tuned on PACS.
| Run | What it is | Role |
|---|---|---|
| ERM | Plain training on the pooled source domains | Baseline, required |
| MixStyle | Mixes per channel feature statistics (style) between images after ResNet layers 1 to 3, during training only | Implemented, the main method |
| DANN | A domain discriminator plus a gradient reversal layer, so the features stop revealing which domain an image came from | Not implemented, to leave time for the analysis and report |
Why MixStyle is the lead: it is about 15 lines with no extra loss or network, it trains stably, it needs no domain labels, and the professor named it in the spec. It fits PACS well because the domains differ mostly in style. Because it can only create styles between the training styles, it was expected to help least on Sketch. Sketch gained the most instead (see Results).
- Choose hyperparameters and the stopping epoch using only the source domains' validation split. Never tune on the held out domain.
- Run each configuration with 3 seeds and report the mean and spread.
- Metrics: ID accuracy, OOD accuracy, the gap between them, and per domain accuracy.
-
Domain probe: freeze the trained network, summarize each channel after every ResNet stage by its mean and std (the style MixStyle mixes), train a linear classifier to tell source images from held out ones, and convert its error into the A distance,
$d_A = 2(1 - 2\epsilon)$ . This shows at which layer each method keeps or removes domain information, compared with the ImageNet weights before fine tuning. - t-SNE of the features, colored by domain and by class, for ERM and each method.
- ID vs OOD accuracy per method and held out domain (table, and a dot plot whose line lengths are the gap).
- Learning curves with Sketch held out, with the seed spread shaded.
- Confusion matrices on held out Sketch, ERM next to MixStyle.
- Domain probe
$d_A$ per layer and method (notebook only, left out of the report for space). - t-SNE plots colored by domain and by class (notebook only).
Accuracy in percent, mean ± standard deviation over 3 seeds, at the epoch with the best source validation accuracy. ID is the source domains' test split; OOD is the held out domain.
| Held out | ERM ID | ERM OOD | ERM gap | MixStyle ID | MixStyle OOD | MixStyle gap |
|---|---|---|---|---|---|---|
| Art | 95.2 ± 0.6 | 76.3 ± 1.8 | 18.9 ± 1.4 | 95.4 ± 0.1 | 81.3 ± 1.1 | 14.1 ± 1.1 |
| Cartoon | 94.3 ± 0.2 | 73.0 ± 1.9 | 21.3 ± 1.7 | 95.0 ± 0.3 | 77.4 ± 0.8 | 17.6 ± 1.1 |
| Photo | 93.9 ± 0.2 | 95.4 ± 0.6 | -1.5 ± 0.5 | 93.9 ± 0.3 | 95.5 ± 0.3 | -1.7 ± 0.6 |
| Sketch | 95.3 ± 0.3 | 64.8 ± 0.6 | 30.5 ± 0.8 | 95.5 ± 0.3 | 72.6 ± 1.8 | 22.9 ± 2.1 |
- ID accuracy is about 95% for both methods; the shift costs ERM up to 30 points on Sketch.
- MixStyle raises OOD accuracy by 4 to 8 points on Art, Cartoon, and Sketch without changing ID accuracy.
- Sketch gains the most (+7.8), the opposite of the plan's expectation that MixStyle would help least there.
- Holding out Photo is barely a shift, because the ImageNet pretraining already covers photos.
- The domain probe shows MixStyle does not make the domains look alike: its A distance is within about 0.05 of ERM's at every layer, and Sketch stays fully separable. The gain comes from the classifier tolerating style changes, not from removing domain information.
- These numbers are within about 3 points of the ResNet-18 results in the MixStyle paper.
- On held out Sketch, ERM predicts "elephant" for 46% of people, 28% of dogs, and 18% of houses (pooled over seeds). MixStyle cuts person to elephant to 20% and house to elephant to 7%. One likely reason: elephant is the gray, low texture class in the source domains, so ERM uses color and texture as a shortcut that fails on sketches. This has not been checked against the source images' color statistics.
Figures are in results/figures/, built by notebooks/analysis.ipynb.
The whole pipeline, from download to report:
flowchart TD
A["scripts/download_pacs.py<br/>data/pacs/domain/class/*.jpg"] --> B["scripts/make_splits.py<br/>splits/pacs.csv: train / val / test per image"]
B --> C["styleshift sweep<br/>2 methods x 4 targets x 3 seeds"]
C -->|"calls run(cfg) 24 times,<br/>skips runs with metrics.json"| D
subgraph D["styleshift train: train.py run()"]
direction TB
D1["config.py: TrainConfig<br/>method, target, seed, lr ..."] --> D2["data.py: split_rows<br/>sources = 3 domains, target = held out"]
D2 --> D3["build_loaders<br/>train | val | id_test | ood_test"]
D3 --> D4["methods.py: build_algorithm<br/>ERM or ERM + MixStyle"]
D4 --> D5["train_epoch<br/>one pass over source train"]
D5 --> D6["evaluate val, ID test, OOD test"]
D6 --> D7{"best val<br/>so far?"}
D7 -->|yes| D8["keep weights and this epoch's numbers"]
D7 -->|no| D9
D8 --> D9{"epoch < 50?"}
D9 -->|yes| D5
D9 -->|no| D10["write history.csv, model.pt, metrics.json<br/>runs/method/target/seedK/"]
end
D --> E["styleshift collect<br/>results/runs.csv, summary.csv, history.csv"]
D --> F["styleshift probe<br/>per layer domain probe + t-SNE<br/>results/probe.csv, results/tsne/"]
D --> K["styleshift confusion<br/>true vs predicted class counts<br/>results/confusion.csv"]
E --> G["notebooks/analysis.ipynb<br/>results/figures/*.png and *.pdf<br/>report/table_id_ood.tex"]
F --> G
K --> G
G --> H["report/main.tex<br/>2 page report"]
One training step with MixStyle. The ERM baseline is the same network without the three MixStyle boxes, which is the only difference between the two methods.
flowchart LR
X["batch of 64 source images<br/>x, class y, domain d"] --> S["stem"]
S --> L1["layer1"] --> M1["MixStyle"] --> L2["layer2"] --> M2["MixStyle"] --> L3["layer3"] --> M3["MixStyle"] --> L4["layer4"]
L4 --> P["avg pool<br/>512 features"] --> C["classifier<br/>7 classes"]
C --> LOSS["cross entropy vs y"] --> BW["backward + SGD step"]
subgraph MS["each MixStyle, training only, 50% of batches"]
direction TB
a["mu, sig per image per channel<br/>= the style"] --> b["normalize: (x - mu) / sig<br/>= content without style"]
b --> c["pick a random partner image<br/>blend: lmda * own + (1 - lmda) * partner"]
c --> e["x_normed * sig_mix + mu_mix<br/>= same content, mixed style"]
end
style_shift_generalization/
README.md
pyproject.toml
scripts/
download_pacs.py fetches PACS into data/pacs/<domain>/<class>/
make_splits.py writes splits/pacs.csv (already committed)
splits/
pacs.csv train, val, or test for every image
src/style_shift_generalization/
__init__.py the styleshift command: train, sweep, collect, probe, confusion
config.py TrainConfig, every setting with its default
data.py PACS dataset, transforms, leave one domain out loaders
mixstyle.py MixStyle module
models.py ResNet-18 with optional MixStyle after layers 1 to 3
methods.py ERM and MixStyle behind one Algorithm interface
train.py one run: train, evaluate each epoch, save the best val epoch
sweep.py every method x held out domain x seed, resumable
collect.py runs/ -> results/runs.csv, summary.csv, history.csv
probe.py domain probe (A distance) and t-SNE
confusion.py true vs predicted class counts on ID and OOD test
tests/ MixStyle behavior and split leakage checks
notebooks/
analysis.ipynb reads results/, draws the report figures
runs/ one folder per run with its checkpoint (not committed)
results/ metrics CSVs and figures (committed)
report/
main.tex the 2 page write up
refs.bib references
table_id_ood.tex results table, written by the notebook
Makefile builds main.pdf
The project uses uv and Python 3.13.
uv sync # creates .venv and installs everything from uv.lock
source .venv/bin/activate # or .venv/bin/activate.fish in fishDownload PACS (about 190 MB) and export it to data/pacs/:
uv run python scripts/download_pacs.pyThe split in splits/pacs.csv is committed, so there is no need to regenerate it.
scripts/make_splits.py rebuilds it identically, and only overwrites it with --force.
Check that PyTorch sees the GPU:
uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"PyTorch comes from the official CUDA 13.0 wheel index, set in pyproject.toml under [tool.uv.sources].
The RTX 5060 is a Blackwell card (compute capability 12.0), which needs a CUDA 12.8 or newer build.
| Package | Used for |
|---|---|
torch |
Models, training, GPU |
torchvision |
ResNet-18 with ImageNet weights, image transforms, ImageFolder loading |
numpy |
Array math, the MixStyle and probe calculations |
pandas |
Collecting results across runs and seeds into tables |
scikit-learn |
Linear domain probe, t-SNE, metrics |
matplotlib |
Figures for the report |
pillow |
Reading the PACS image files |
tqdm |
Progress bars during training and export |
huggingface_hub |
Downloading the pinned PACS parquet file |
pyarrow |
Reading the parquet file |
Dev only (uv sync installs these too):
| Package | Used for |
|---|---|
ruff |
Lint and format |
pytest |
Tests, for example that MixStyle leaves |
ipykernel |
Running exploration notebooks against this venv |
Each run trains one method with one held out domain and one seed:
uv run styleshift train --method erm --target sketch --seed 0
uv run styleshift train --method mixstyle --target sketch --seed 0Every setting in src/style_shift_generalization/config.py is also a flag, for example --epochs 1 for a quick check.
uv run styleshift train --help lists them all.
A run writes config.json, history.csv, model.pt, and metrics.json to runs/<method>/<target>/seed<k>/.
The full experiment is 2 methods x 4 held out domains x 3 seeds, 24 runs of about 8 minutes each on an RTX 5060, about 3 hours 20 minutes in total:
uv run styleshift sweep # skips any run whose metrics.json already exists, so it resumes after a stop
uv run styleshift collect # writes results/runs.csv, summary.csv, history.csv
uv run styleshift probe # writes results/probe.csv and results/tsne/
uv run styleshift confusion # writes results/confusion.csvuv run styleshift sweep --methods mixstyle --targets sketch --seeds 0 1 2 runs a subset.
Then open notebooks/analysis.ipynb and run all cells to build the table and figures in results/figures/ and the LaTeX table in report/.
The report is a two column LaTeX paper, at most 2 pages before the references, and needs pdflatex and bibtex (TeX Live).
The built PDF is not committed; the submitted PDF links back to this repo.
cd report && make # writes report/main.pdf
make clean # removes build files and the PDFRed TODO text marks sections still to be written.
Numbers are never typed into main.tex: the table comes from the notebook and the figures from results/figures/*.pdf, so rerunning the notebook updates the report.
uv run pytest
uv run ruff check . && uv run ruff format --check .- GPU: NVIDIA RTX 5060 (8 GB), driver supporting CUDA 13.4.
- Verified:
torch 2.14.0+cu130detects the GPU and runs a ResNet-18 forward pass. - Each run took about 8 minutes, and the full 24 run sweep about 3 hours 20 minutes.
This repo will be public and linked in the submission.
- Do not commit the PACS data.
data/is in.gitignore, and the download script recreates it. - Do not commit model checkpoints.
- Do not commit the course slides or screenshots of them. Link to the course instead.
- Read any API keys or tokens (for example, experiment tracking) from environment variables. Never commit them.
- Keep the license notices of any adapted code.
Code this project draws on:
- DeepDG, the course's recommended domain generalization toolkit, which includes ERM, DANN, CORAL, and loaders for PACS.
- MixStyle official implementation.
- DomainBed, which includes a PACS download script and the standard evaluation protocol.
Papers:
- Zhou et al., Domain Generalization with MixStyle, ICLR 2021. arXiv 2104.02008
- Ganin et al., Domain-Adversarial Training of Neural Networks, JMLR 2016. arXiv 1505.07818
- Li et al., Deeper, Broader and Artier Domain Generalization (the PACS dataset), ICCV 2017. arXiv 1710.03077
- Gulrajani and Lopez-Paz, In Search of Lost Domain Generalization, ICLR 2021. arXiv 2007.01434
- Wang et al., Generalizing to Unseen Domains: A Survey on Domain Generalization, TKDE 2022. arXiv 2103.03097

