[AUTO] Include the CPU for cache warmup and update doc by peterchen-intel · Pull Request #39561 · openvinotoolkit/openvino

peterchen-intel · 2026-11-05T23:11:11Z

Summary

Follow-up to #39407 and the documentation review.

Include eligible CPU devices in ov::intel_auto::compile_for_all cache warm-up instead of filtering them out in the plugin, scheduler, and worker. Continue to exclude the final execution device and devices removed after failed startup compilation.
Keep the existing CPU-candidate preservation logic for cache hits and disabled fallback settings. Add regression coverage for CPU warm-up and distinguish it from the CPU startup helper.
Move detailed documentation into a dedicated subsection with Python/C++ examples. Align the public property comments and C API value type, explain discovery for plain AUTO, cache prerequisites and ignore rules, conditional fallback disabling, later-cache reuse, and background memory/teardown costs.
Preserve the public property name and default (false). This is cache population for later compilation, not automatic execution on every device or a device-switching operation.

Tickets:

CVS-191307

Scope and risks

When CPU is an additional candidate, enabling compile_for_all now disables the CPU startup helper and runtime inference fallback, consistently with the existing warm-up policy. First-inference latency and background memory/compilation cost can increase. Warm-up remains best-effort and teardown waits for compilation already in flight.

No changes to Core cache identity, C/Python binding implementation, or public property naming.

AI Assistance

AI assistance used: yes.

Copilot

🟢 Approval recommended

No active review issues are reported for the extension of AUTO cache warm-up to eligible CPU candidates.

No open issues 🎉

What changed in this PR

This follow-up extends AUTO’s compile_for_all cache warm-up to eligible CPU candidates, preparing caches for later compilation without switching the current execution device. The property name and default (false) remain unchanged.

Changes:

Remove CPU exclusions from candidate validation, scheduling, and background warm-up while still excluding the final execution device.
Add regression coverage for CPU warm-up, startup-helper separation, and subsequent cache reuse.
Document cache requirements, fallback behavior, resource costs, and Python/C++ usage.

File	Description
`src/plugins/auto/tests/unit/cache_compile_for_all_test.cpp`	Add CPU warm-up cases and update compilation-count expectations.
`src/plugins/auto/tests/functional/behavior/caching_test.cpp`	Verify CPU cache creation and later reuse alongside other devices.
`src/plugins/auto/src/plugin.cpp`	Include CPU in warm-up candidate validation.
`src/plugins/auto/src/cache_warmup.cpp`	Allow background warm-up for non-selected CPU candidates.
`src/plugins/auto/src/auto_schedule.cpp`	Include CPU when deciding whether warm-up disables fallbacks.
`src/inference/include/openvino/runtime/auto/properties.hpp`	Clarify the public property’s behavior and requirements.
`src/bindings/c/include/openvino/c/auto/properties.h`	Document C API boolean values and fallback effects.
`src/bindings/c/docs/api_overview.md`	Add spacing between property declarations.
`docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection.rst`	Add a dedicated warm-up subsection covering usage and trade-offs.
`docs/articles_en/assets/snippets/ov_auto.py`	Add a Python cache warm-up example.
`docs/articles_en/assets/snippets/AUTO0.cpp`	Add a C++ example, required header, and copyright notice.

_{Review effort: Balanced}

wangleis

Thank you for the updates. While reviewing the latest PR changes, we identified a few points regarding documentation consistency, snippet executability, and background CPU warm-up resource considerations. Detailed inline comments are provided below.

wangleis · 2026-11-06T03:02:48Z

    std::vector<DeviceInformation> devices;
    for (const auto& device : candidates) {
-        if (device.device_name != actual_device && ov::DeviceIDParser(device.device_name).get_device_name() != "CPU") {
+        if (device.device_name != actual_device) {


Now that CPU candidates are included in background cache warm-up (removing the previous CPU exclusion here and in AutoSchedule), please note that the class documentation in src/plugins/auto/src/cache_warmup.hpp (line 22) still says:

// 1. start() is called once the actual device is known. It filters out the actual device and CPU, then hands this // object to CacheWarmupManager (once per object).

Please update cache_warmup.hpp to remove "and CPU" so that the header comments remain consistent with this change.

wangleis · 2026-11-06T03:02:49Z

+def part8():
+    #! [compile_for_all]
+    core = ov.Core()
+    model = core.read_model("model.xml")


Two minor issues with this snippet:

part8() is defined here but never called in main(), so this snippet is not validated when running ov_auto.py.

Unlike the other snippet functions in this file which use the shared get_model() helper from utils.py, part8() tries to load "model.xml" directly via core.read_model("model.xml"), which will raise FileNotFoundError if executed.

Please consider updating part8() to use get_model() and a temporary cache directory, and invoke part8() within the GPU-available check block in main().

wangleis · 2026-11-06T03:02:49Z

+* Skips devices with caching disabled or a single-file cache path (``*.bin``). Warm-up failures are logged as warnings.
+  Core reuses compatible cache blobs and may rebuild invalid ones.
+
+Warm-up compiles one device at a time and releases each temporary compiled model. Background compilation adds


Regarding background compilation overhead and resource usage:
When warming up GPU, make_warmup_config configures ov::intel_gpu::hint::host_task_priority(ov::hint::Priority::LOW). However, the CPU plugin does not support host_task_priority or compilation_num_threads. As a result, background warm-up on CPU runs with full TBB thread concurrency, competing for host CPU cores and memory bandwidth with active inference on the primary execution device.

It would be helpful to explicitly mention in this section that background CPU warm-up can cause host CPU and memory contention during ongoing inference (or consider applying a reduced-stream / lower-thread configuration in make_warmup_config for CPU, since CPU cache key generation depends only on full_name).

- Drop stale "and CPU" from the CacheWarmup class comment - Run the compile_for_all Python snippet from main() with get_model() and a temporary cache dir - Document host CPU/memory contention of background CPU warm-up - Remove the redundant candidate-count check in plugin.cpp; AutoSchedule::init owns it - Trim the public property comments to the contract - Revert the functional test change that does not exercise CPU - Make disablesStartupAndRuntimeFallback distinguish warm-up from the CPU startup helper

Copilot

🟡 Changes recommended

The published compilation fragment uses an undefined cache_dir, causing NameError when copied, and removes the temporary cache directory before releasing the compiled model, potentially racing background writes.

1 open issue

Define cache_dir in the published compile_for_all fragment · New

_{Review effort: Balanced}

+            device_name="AUTO",
+            config={
+                device.priorities: "GPU,CPU",
+                properties.cache_dir: cache_dir,


peterchen-intel requested review from a team as code owners November 5, 2026 23:11

peterchen-intel requested review from mfhansen and removed request for a team November 5, 2026 23:11

peterchen-intel requested review from wgzintel and a balanced review from Copilot November 6, 2026 00:12

peterchen-intel assigned wangleis Nov 6, 2026

Copilot started reviewing on behalf of peterchen-intel November 6, 2026 00:13

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

peterchen-intel mentioned this pull request Nov 6, 2026

[AUTO] Create model cache for all devices #39407

Merged

wangleis reviewed Nov 6, 2026

View reviewed changes

peterchen-intel force-pushed the fix/auto-cpu-cache-warmup-doc branch from 8d212d5 to 1da41d5 Compare November 6, 2026 03:46

peterchen-intel added do_not_review do_not_merge labels Nov 6, 2026

peterchen-intel added 6 commits November 6, 2026 15:35

[AUTO] Include the CPU for cache warmup and update doc

672c8b1

[AUTO] Add required copyright header to documentation snippet

dfcde9c

[AUTO] Fix cache warmup documentation and test formatting

685d30c

[AUTO] Simplify cache warmup documentation

5d1cc8d

[AUTO] Accept %zu in AUTO plugin log format check

15d9690

peterchen-intel force-pushed the fix/auto-cpu-cache-warmup-doc branch from 1da41d5 to 6e3aa46 Compare November 6, 2026 07:47

peterchen-intel requested a review from wangleis November 6, 2026 10:14

peterchen-intel requested a balanced review from Copilot November 6, 2026 10:14

peterchen-intel removed the do_not_review label Nov 6, 2026

Copilot started reviewing on behalf of peterchen-intel November 6, 2026 10:15

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread docs/articles_en/assets/snippets/ov_auto.py

device_name="AUTO",

config={

device.priorities: "GPU,CPU",

properties.cache_dir: cache_dir,

Navigation Menu

Repository navigation

[AUTO] Include the CPU for cache warmup and update doc - #39561

[AUTO] Include the CPU for cache warmup and update doc#39561
peterchen-intel wants to merge 6 commits into
openvinotoolkit:masterfrom
peterchen-intel:fix/auto-cpu-cache-warmup-doc

peterchen-intel commented Nov 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

wangleis left a comment

Uh oh!

wangleis Nov 6, 2026

Uh oh!

wangleis Nov 6, 2026

Uh oh!

wangleis Nov 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

peterchen-intel commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tickets:

Scope and risks

AI Assistance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

wangleis left a comment

Choose a reason for hiding this comment

Uh oh!

wangleis Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

wangleis Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

wangleis Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

peterchen-intel commented Nov 5, 2026 •

edited

Loading