Skip to content

fix: self-heal stale annots.pkl cache instead of crashing with KeyError - #1911

Closed
HikeAndMap wants to merge 2 commits into
Megvii-BaseDetection:mainfrom
HikeAndMap:fix/stale-annots-cache
Closed

HikeAndMap wants to merge 2 commits into
Megvii-BaseDetection:mainfrom
HikeAndMap:fix/stale-annots-cache

Conversation

@HikeAndMap

Copy link
Copy Markdown

voc_eval() builds a ground-truth annotation cache (annots.pkl) the first time eval runs for an experiment, then trusts it unconditionally on every subsequent eval - there's no check that it still matches the current imagesetfile/dataset. Iterating on a custom dataset (adding images, re-labeling, re-tiling) between training runs without manually deleting <data_dir>/annotations_cache/ causes the next eval to crash with KeyError as soon as it hits an image that wasn't present when the cache was written.

This is very likely the real, undiagnosed cause behind several long-open upstream issues that all report the same symptom without ever tracing it to a stale pickle cache (#1397, #874, #1490).

Fixed by checking every imagesetfile entry is a key in the loaded recs dict; if any are missing, rebuild the cache from the XML annotations instead of raising. Verified against a real dataset that hit this exact crash - eval now rebuilds the cache and completes normally, and stays fast (no rebuild) on subsequent evals once back in sync.

luxedo and others added 2 commits June 8, 2025 11:52
voc_eval() builds a ground-truth annotation cache (annots.pkl) the first
time eval runs for an experiment, then trusts it unconditionally on every
subsequent eval - there's no check that it still matches the current
imagesetfile/dataset. Iterating on a custom dataset (adding images,
re-labeling, re-tiling) between training runs without manually deleting
<data_dir>/annotations_cache/ causes the next eval to crash with
KeyError as soon as it hits an image that wasn't present when the cache
was written.

This is very likely the real, undiagnosed cause behind several long-open
upstream issues that all report the same symptom without ever tracing it
to a stale pickle cache (Megvii-BaseDetection#1397, Megvii-BaseDetection#874, Megvii-BaseDetection#1490).

Fixed by checking every imagesetfile entry is a key in the loaded recs
dict; if any are missing, rebuild the cache from the XML annotations
instead of raising. Verified against a real dataset that hit this exact
crash - eval now rebuilds the cache and completes normally, and stays
fast (no rebuild) on subsequent evals once back in sync.
@CLAassistant

CLAassistant commented Sep 2, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ luxedo
❌ HikeAndMap


HikeAndMap seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@HikeAndMap HikeAndMap closed this Sep 11, 2026
@HikeAndMap
HikeAndMap force-pushed the fix/stale-annots-cache branch from 96ed3f3 to 3cf33c1 Compare September 11, 2026 06:36
@HikeAndMap
HikeAndMap deleted the fix/stale-annots-cache branch October 18, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants