[yangwang201911]

This pull request introduces a new "dynamic device selection" mode to the AUTO plugin, allowing per-inference device selection based on resource-aware properties. The changes add infrastructure to serialize inference requests, queue and dispatch them as devices are re-selected, and ensure models are compiled on-demand for target devices. The implementation also handles device failures and ensures proper resource cleanup.

Details:

• Added the m_dynamic_device_selection flag to ScheduleContext, enabling per-inference device selection when resource-aware properties are set.
• In AutoSchedule, added members and logic to serialize inference requests, queue them if a device is busy, and re-select or recompile models for each inference as needed.
• Modified schedule_to_worker_infer_request to use dynamic scheduling when enabled, ensuring requests are queued and dispatched properly.
• In generate_workers, limited the number of worker requests per device to one in dynamic mode to guarantee serialized execution and safe device switching.
• Ensured that models are kept alive for on-demand compilation, and added detailed logging for device selection, compilation, and device switching events to aid debugging and profiling.

Tickets:

• CVS-193488

AI Assistance:

• AI assistance used: yes
• If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

[Copilot]

🟡 Changes recommended

Critical stateful-model and retry-serialization issues remain, along with resource and recovery cleanup issues.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR adds resource-aware, per-inference device selection to AUTO with serialized dispatch and on-demand model compilation.

Changes:

• Adds dynamic scheduling, queuing, device reselection, and execution gating.
• Adds on-demand model compilation and caching with one worker per device.
• Adds scheduling integration and dynamic-selection unit tests.

File summaries

File	Summary and final review notes
src/plugins/auto/tests/unit/dynamic_device_selection_test.cpp	Adds dynamic-selection unit coverage.
src/plugins/auto/src/schedule.hpp	Adds scheduling exception and execution-slot interfaces.
src/plugins/auto/src/schedule.cpp	Critical (1 vote): retry handling can release the execution gate too early, allowing concurrent execution.
src/plugins/auto/src/plugin.cpp	Critical (3 votes): dynamic switching can produce incorrect stateful-model behavior.
src/plugins/auto/src/common.hpp	Adds dynamic-selection context state.
src/plugins/auto/src/auto_schedule.hpp	Declares dynamic scheduler state and helpers.
src/plugins/auto/src/auto_schedule.cpp	Moderate (2 votes): fallback registrations may remain in the process-wide priority map. Moderate (1 vote): failed devices retain compiled models and worker requests. Nit (1 vote): concurrent queueing is not exercised. Moderate (1 vote): cache publication before worker creation can prevent recovery. Moderate (1 vote): CPU fallback reuse can skip required compilation.

Review details

Suppressed comments (4)

src/plugins/auto/src/auto_schedule.cpp:30

• [HIGH] Removing a failed device from m_device_priorities leaves its dynamic compiled model and worker request resident: m_dynamic_compiled_models owns the SoCompiledModel, while m_worker_requests owns the device infer request. Every runtime-failed device therefore keeps its device resources until the whole AUTO compiled model is destroyed, even though it can no longer be selected. Defer safe removal until the current callback has released the worker, then reclaim these entries.

                m_context->m_device_priorities.erase(iter);
                LOG_WARNING_TAG("[dynamic] inference failed on device:%s, exclude it and retry on another device",

src/plugins/auto/src/auto_schedule.cpp:547

• [HIGH] The new queue path is not exercised by the tests: every helper call to infer() waits for completion before starting the next request, so m_gate_pending_tasks is never populated. A regression in concurrent queuing or release_execution_slot() would therefore pass this suite; add a test that starts multiple async requests concurrently and waits for all of them.

            m_gate_pending_tasks.emplace_back(std::move(pipeline_task), preferred_device);
            LOG_DEBUG_TAG("[dynamic] an inference is still running, request queued, queue size:%ld",
                          static_cast<long>(m_gate_pending_tasks.size()));

src/plugins/auto/src/auto_schedule.cpp:652

• [HIGH] The cache entry is published before generate_workers() succeeds. If creating the first worker throws, dispatch reports the error but leaves this device marked as compiled; later requests skip compilation and call run_pipeline_task() with no idle worker, so the device can never recover. Publish the compiled model only after worker creation succeeds, or erase the entry on the failure path.

    m_dynamic_compiled_models[device.device_name] = context.m_compiled_model;
    generate_workers(device.device_name, context.m_compiled_model);

src/plugins/auto/src/auto_schedule.cpp:643

• [HIGH] The new on-demand compile path can fail to use an available CPU fallback. Dynamic mode explicitly disables m_startup_fallback, leaving m_compile_context[CPU].m_is_enabled false; try_to_compile_model() still performs its CPU-config equality shortcut without checking that the CPU context is enabled/already compiled. If the selected accelerator fails and the candidate CPU has the same (for example, empty) config as that default context, the helper returns before compiling CPU and ensure_device_ready() reports a scheduling error. Guard that reuse shortcut with the CPU context state or provide a dedicated on-demand fallback path.

    try_to_compile_model(context, m_dynamic_model ? m_dynamic_model->clone() : nullptr);

• Files reviewed: 7/7 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Unresolved critical, blocker, and moderate findings affect device safety, fallback behavior, caching, cleanup, and stateful models.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

src/plugins/auto/src/auto_schedule.cpp:640

• [HIGH] The on-demand compile attempt uses a private copy of m_context->m_device_priorities; when a target such as GPU.1 fails compilation, try_to_compile_model removes it only from that temporary list and can fall back to CPU, but the next inference can select GPU.1 again and pay the full compile/fallback cost repeatedly. Record permanently failed compile candidates in the dynamic scheduler's shared selection state (or cache them as unavailable) so device-failure handling does not become per-inference recompilation.

src/plugins/auto/src/auto_schedule.cpp:648

• [HIGH] The cleanup below is reached only when context.m_is_load_success is true. If the selected device and every fallback fail to compile, try_to_compile_model can leave its last select_device registration in the process-wide priority map, and this return keeps that reservation alive for the scheduler lifetime; lower-priority AUTO instances can then be prevented from selecting the device. Release every on-demand registration on failure as well, preferably by tracking registration ownership with an RAII guard.

    if (!context.m_is_load_success) {
        LOG_WARNING_TAG("[dynamic] compiling the model on device:%s failed, %s",
                        device.device_name.c_str(),
                        context.m_err_message.c_str());
        return false;

src/plugins/auto/src/auto_schedule.cpp:115

• [HIGH] This does not retain a model for path-based compilation. In compile_model_impl, m_context->m_model is null when m_model_path is used, so this assignment leaves m_dynamic_model empty and ensure_device_ready() later calls compile_model(m_context->m_model_path, ...) after the original compile has returned. If the caller removes or loses access to the file, the first device switch fails even though the model was compiled successfully; materialize/retain the model for dynamic mode or disable dynamic selection for path-based compilation.

        // the model has to stay alive to be compiled on demand once the target device changes
        m_dynamic_model = m_context->m_model;
        m_dynamic_executor = m_plugin->get_executor_manager()->get_idle_cpu_streams_executor(
            ov::threading::IStreamsExecutor::Config{"AutoDynamicSchedule", 1, 0});

• Files reviewed: 8/8 changed files
• Comments generated: 4
• Review effort level: Lite

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical configuration issues and moderate dynamic-compilation cleanup issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

• Disable or reject buffer binding in dynamic scheduling · New
• Disable dynamic mode when device buffer binding is enabled · New

Resolved since last review (4)

• [BLOCKER] Stateful-model detection is skipped for path-based compilation: the model_path branch…
• [HIGH] The newly compiled model is cached before generate_workers() succeeds. If worker creation…
• [HIGH] The per-inference selection registers a device through select_device() and immediately…
• [HIGH] A non-empty preferred_device is used for remote tensors and is explicitly treated as…

Previously missed (1)

In code that hasn't changed since last review

Exclude failed dynamic devices from subsequent selection

src/​plugins/​auto/​src/​auto_schedule.cpp:655

[HIGH] A compile failure on a dynamically selected device is only removed from the local context.m_meta_devices; m_context->m_device_priorities still contains it. If resource selection picks that device again, this path recompiles the already cached fallback and calls generate_workers again for the same device, leaving duplicate/stale entries in its idle-worker queue and repeating the compile on every such inference. Keep failed compile targets out of subsequent dynamic selection, or reuse an existing cached fallback without regenerating its workers.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved moderate issues remain in fallback worker reuse and priority cleanup, and tests do not cover concurrent queueing and key fallback paths.

Review effort: Lite
Findings: None

Resolved since last review (2)

• Disable dynamic mode when device buffer binding is enabled
• Disable or reject buffer binding in dynamic scheduling

Previously missed (2)

In code that hasn't changed since last review

Reuse cached worker after device fallback

src/​plugins/​auto/​src/​auto_schedule.cpp:656

[HIGH] When a newly selected device fails compilation, try_to_compile_model() can switch context.m_device_info to a device already present in m_dynamic_compiled_models (for example, a later GPU selection falling back to the initial CPU). This path does not consult that cache: for an initial CPU, the legacy m_compile_context[CPU] is disabled, so the fallback is either reported as unsuccessful or generate_workers() is called again on the existing one-entry idle queue, causing the request to fail instead of reusing the cached worker. After fallback, check the dynamic cache and dispatch its existing worker/model without recompiling or regenerating workers.

Add concurrent test for queued async requests

src/​plugins/​auto/​tests/​unit/​dynamic_device_selection_test.cpp:80

[HIGH] This helper calls infer() serially, so no test ever enters schedule_dynamic_task() while m_gate_busy is true. The new queue and release_execution_slot() handoff are the core of concurrent request support; regressions that drop or mis-handle queued requests will pass all of these tests. Add a concurrent start_async() test with a blocked worker and verify that every queued request completes.

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical dispatch-gate and stateful-model issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 2

Open (4)

• Failed pipeline dispatch leaves execution gate permanently busy · New
• Stateful single-target models incorrectly use dynamic mode · New
• Failed device compilation is retried on every inference · New
• Test insert fails to replace configured device priorities · New

[Copilot]

🟡 Changes recommended

Unresolved fallback reservation cleanup, low-power remote-tensor rejection, and stale execution-device reporting affect correctness.

3 open issues

• Prevent double-unregister from deleting another model's reservation · New
• Report the active device after dynamic dispatch switches · New
• Resolve out-of-list low-power devices for remote tensors · New

3 resolved since last review

• Reject AUTO as a low-power target before parsing
• Prevent double removal of another model's device reservation
• Disable failed low-power CPU override before early return

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Unresolved stateful-model safety, fallback handling, registration cleanup, and test setup defects prevent approval.

3 open issues

• Disable fallback modes for stateful single-candidate models · New
• Reinitialize GPU mock after replacing the model · New
• Reset priority release state after fallback selection · New

2 previously missed issues

In code that hasn't changed since last review

Allow fallback selection for out-of-list low-power targets

src/​plugins/​auto/​src/​auto_schedule.cpp:59

[HIGH] With THROUGHPUT, priorities GPU.0,CPU, and LOW_POWER_DEVICE=GPU.1, initial selection can choose GPU.1 while dynamic scheduling is disabled. If GPU.1 inference fails, the candidate lookup misses it and returns false without trying the configured runtime fallback devices. Allow fallback selection for this out-of-list target while preserving the protection against repeated callbacks starting another fallback.

Reuse cached fallback models before recompiling

src/​plugins/​auto/​src/​auto_schedule.cpp:453

[MEDIUM] If a new target fails compilation and fallback selects an already-cached device, try_to_compile_model() recompiles that device before ensure_device_ready() checks the cache again. This adds compilation cost and can fail the inference despite an available cached model and worker. Reuse the cached model here before compiling again, while preserving the registration cleanup.

3 resolved since last review

• Prevent double-unregister from deleting another model's reservation
• Resolve out-of-list low-power devices for remote tensors
• Report the active device after dynamic dispatch switches

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Deferred compilation retains a caller-mutable model, allowing later device switches to compile a different graph from the initial device.

1 open issue

• Clone model to preserve compilation-time graph snapshot · New

3 resolved since last review

• Reinitialize GPU mock after replacing the model
• Disable fallback modes for stateful single-candidate models
• Reset priority release state after fallback selection

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Confirmed selection-ordering and fallback-cache issues remain, and the asynchronous scheduling changes also warrant human review.

3 open issues

• Clone model to preserve compilation-time graph snapshot
• Reuse cached model before recompiling after GPU fallback · New
• Apply low-power device override before empty-candidate fallback · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Exhausting ordinary candidates blocks subsequent inference even when a healthy low-power device becomes eligible again.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Allow low-power fallback when ordinary candidates are exhausted

src/​plugins/​auto/​src/​auto_schedule.cpp:689

[HIGH] With priorities GPU.0 and LOW_POWER_DEVICE=CPU, compiling in low-power mode and then failing GPU.0's first compilation outside that mode removes the sole ordinary candidate. This assertion then rejects every unpinned inference, even after low-power mode resumes with a healthy cached CPU model. Allow the available low-power target to be selected when ordinary candidates are exhausted, without calling select_device() with an empty list, and add a regression test for this transition.

3 resolved since last review

• Clone model to preserve compilation-time graph snapshot
• Apply low-power device override before empty-candidate fallback
• Reuse cached model before recompiling after GPU fallback

_{Review effort: Balanced}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues

• This sets m_this_scheduling_exception / m_this_worker_infer_request as thread_local and then… · New
• Dynamic scheduling reports failures via Schedule::m_this_scheduling_exception (thread_local).… · New
• m_select_device_count is mutated from the dynamic scheduling thread (inside the select_device… · New
• m_select_device_count is mutated from the dynamic scheduling thread (inside the select_device… · New

_{Review effort: Lite}

[Copilot]

🔵 Needs a closer look

Cross-thread scheduling, fallback, and teardown require targeted runtime validation and maintainer review, with a compilation-exclusion issue still unresolved.

5 open issues

• Dynamic scheduling reports failures via Schedule::m_this_scheduling_exception (thread_local).…
• This sets m_this_scheduling_exception / m_this_worker_infer_request as thread_local and then…
• Exclude all failed devices from subsequent compilation retries · New
• m_select_device_count is mutated from the dynamic scheduling thread (inside the select_device…
• m_select_device_count is mutated from the dynamic scheduling thread (inside the select_device…

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

A critical use-after-free remains possible when a failed dispatch completes the request before worker cleanup finishes, requiring scheduler lifetime protection for both dispatch submission paths.

1 open issue

• Prevent use-after-free during asynchronous worker cleanup · New

1 previously missed issue

In code that hasn't changed since last review

Handle first failure for out-of-list low-power fallback target

src/​plugins/​auto/​src/​auto_schedule.cpp:59

With THROUGHPUT, priorities GPU.0,CPU, LOW_POWER_DEVICE=GPU.1, and startup fallback disabled, GPU.1 can be selected initially but its inference failure never tries a fallback despite runtime fallback being enabled. Because GPU.1 is absent from m_device_priorities, the lookup below treats its first failure as an already-handled failure and returns false. Handle the first failure of an out-of-list low-power target as a fallback transition, while preserving the duplicate-failure guard for subsequent worker callbacks.

5 resolved since last review

• Dynamic scheduling reports failures via Schedule::m_this_scheduling_exception (thread_local).…
• This sets m_this_scheduling_exception / m_this_worker_infer_request as thread_local and then…
• Exclude all failed devices from subsequent compilation retries
• m_select_device_count is mutated from the dynamic scheduling thread (inside the select_device…
• m_select_device_count is mutated from the dynamic scheduling thread (inside the select_device…

_{Review effort: Balanced}