Skip to content

fix(providers): treat an env-selected Bedrock provider as an available LLM - #527

Open
arniecommits wants to merge 3 commits into
NVIDIA:mainfrom
arniecommits:fix/bedrock-provider-authoritative-in-llm-availability
Open

arniecommits wants to merge 3 commits into
NVIDIA:mainfrom
arniecommits:fix/bedrock-provider-authoritative-in-llm-availability

Conversation

@arniecommits

Copy link
Copy Markdown

The problem

With SKILLSPECTOR_PROVIDER=bedrock and working AWS credentials, the three semantic_* analyzers are silently dropped from the graph:

WARNING [skillspector.graph] Skipping analyzer semantic_developer_intent: required API key is missing
WARNING [skillspector.graph] Skipping analyzer semantic_quality_policy: required API key is missing
WARNING [skillspector.graph] Skipping analyzer semantic_security_discovery: required API key is missing

…while meta_analyzer and mcp_tool_poisoning run on Bedrock in the same process and bill real tokens. The scan still exits successfully, so the only sign that detection was halved is a warning on stderr.

Root cause

create_graph() drops any analyzer with requires_api_key = True unless is_llm_available() is true. That function probes the real chat model only when has_provider_binding() is true — and that reports whether a provider was injected through use_provider(), which only the programmatic/MCP path does. Choosing a provider with SKILLSPECTOR_PROVIDER, the only way available from the CLI, leaves the ContextVar empty, so the check falls through to _resolve_llm_credentials(), which understands API keys and knows nothing about the boto3 credential chain.

Reproduced directly:

>>> os.environ["SKILLSPECTOR_PROVIDER"] = "bedrock"
>>> type(get_active_provider()).__name__
'BedrockProvider'
>>> has_provider_binding()
False
>>> is_llm_available()
(False, 'No LLM API key configured. ...')

BedrockProvider.resolve_credentials() returns None by design — its own docstring notes that the OpenAI fallback "is irrelevant for Bedrock" — so no API-key path can ever see it.

_resolve_default_chat_model() has the same shape and falls through to OpenAI for the same reason.

The fix

get_model_config_provider() already gets this right: it treats an explicit binding, a CLI provider, or Bedrock as authoritative, and its docstring says as much. This extracts that predicate as provider_is_authoritative() and uses it at the two sites in llm_utils.py that lack it, so the three cannot drift apart.

Nothing changes for other providers: CLI providers are still short-circuited earlier, and an unbound API-key provider is still judged on its credentials. A Bedrock setup with no working credentials now fails the model probe rather than the API-key lookup, so the reported reason is the provider's own instead of a misleading "no API key".

Verification

On a real scan of the same skill, SKILLSPECTOR_PROVIDER=bedrock:

before after
semantic_* in analyzer_statuses none all three
metadata.llm_available false true
metadata.llm_calls_succeeded 2/2 5/5
metadata.meta_analysis_applied false true
findings 8 16

Five unit tests added in tests/unit/test_llm_utils.py, covering both directions — that Bedrock is authoritative without an API key, that an unbound API-key provider still is not, and that a Bedrock provider which cannot build a model reports its own error.

tests/unit tests/provider tests/nodes run identically before and after (12 pre-existing environment-dependent failures in both; 3803 → 3808 passing, the five new tests).

🤖 Generated with Claude Code

Arnab Roy and others added 2 commits September 11, 2026 10:43
…e LLM

`is_llm_available()` decides whether `create_graph()` wires the analyzers
that set `requires_api_key = True` — the three `semantic_*` ones. It
probes the real chat model only when `has_provider_binding()` is true,
and that reports whether a provider was injected via `use_provider()`,
which only the programmatic/MCP path does. Selecting a provider with
`SKILLSPECTOR_PROVIDER` — the only way from the CLI — leaves that
ContextVar empty, so the check falls through to `_resolve_llm_credentials()`,
which understands API keys and not the boto3 credential chain.

With `SKILLSPECTOR_PROVIDER=bedrock` and working AWS credentials the
result is:

    get_active_provider()   -> BedrockProvider
    has_provider_binding()  -> False
    is_llm_available()      -> (False, "No LLM API key configured...")

so all three semantic analyzers are dropped, while `meta_analyzer` (wired
outside that loop) and `mcp_tool_poisoning` run on Bedrock in the same
process and bill real tokens. The availability check disagrees with the
model path it is checking. `_resolve_default_chat_model()` has the same
shape and falls through to OpenAI for the same reason.

`get_model_config_provider()` already gets this right — it treats an
explicit binding, a CLI provider, or Bedrock as authoritative, and its
docstring says so. This extracts that predicate as
`provider_is_authoritative()` and uses it at the two sites in
`llm_utils.py` that lack it, so the three cannot drift.

Behaviour for other providers is unchanged: CLI providers are still
short-circuited earlier, and an unbound API-key provider is still judged
on its credentials. A Bedrock setup with no working credentials now fails
the model probe instead of the API-key lookup, reporting the provider's
own reason rather than a misleading "no API key".

Verified on a real scan with `SKILLSPECTOR_PROVIDER=bedrock`: before, the
report lists no `semantic_*` entries in
`analysis_completeness.analyzer_statuses`, `llm_available` is false and
`meta_analysis_applied` is false; after, all three run, `llm_calls`
goes 2/2 -> 5/5, `meta_analysis_applied` is true, and the same skill
yields 16 findings instead of 8.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Arnab Roy <alice@cybereducad.com>
`has_provider_binding` is no longer referenced in `llm_utils.py` — both
call sites now go through `provider_is_authoritative` — so the import
tripped F401. Two of the new tests also needed ruff's formatting.

`make lint` and `make format-check` both pass locally now, and
tests/unit/test_llm_utils.py and tests/unit/test_bedrock_provider.py are
66 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Arnab Roy <alice@cybereducad.com>
SanHsien added a commit to SanHsien/SkillSpector that referenced this pull request Sep 11, 2026
…VIDIA#524

Raise reviewed_pr_through to 527 and reviewed_issue_through to 524 in
tools/upstream_baseline.json (commit axis unchanged at 69dcdfb). Every
item gets a verdict in docs/DECISIONS.md: NVIDIA#493/NVIDIA#507/NVIDIA#508/NVIDIA#511 verified
via git merge-base --is-ancestor as already included through the
2.11.1/2.11.2 sync (including NVIDIA#521, which merged only into the still-
open NVIDIA#516 stack, not main); the remaining 27 items stay "wait for
upstream merge", none adopted now.

Two items get dedicated comparison notes per docs/DIVERGENCE.md's
static_runner.py and scripts/compare_scan_accuracy.py rows: NVIDIA#522 uses a
different env var name and different default/semantics than this
fork's SKILLSPECTOR_MAX_STATIC_SECONDS, so merging it cannot simply
delete the divergence row and needs a downstream env var migration
first; NVIDIA#490 extends this fork's own upstream PR NVIDIA#486 with a Python
3.14/POSIX edge case the fork's Windows environment does not hit, so
NVIDIA#486 is left untouched pending upstream's own resolution. NVIDIA#501-NVIDIA#505 and
NVIDIA#518 are also flagged as near-verbatim matches to this fork's existing
Windows test divergence rows, worth revisiting for row deletion once
merged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: SanHsien <34234698+SanHsien@users.noreply.github.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Reviewed head f84febb08d53985305bf8131e48ae550db6e7d37 — APPROVE.

The change consistently reuses the provider-authority predicate for model selection and availability, preserves OpenAI fallback behavior for unbound API-key providers, and adds focused Bedrock success/failure coverage. I found no required code, test, documentation, or security changes.

Merge gate: required lint, unit, and DCO checks pass, but GitHub currently reports mergeStateStatus=BEHIND; update against current main and re-run required checks before merging.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Corrective re-review of current head 794edf7de7e5104e0b5b6c1b7a2cad8aa56f79f6. A mixed-credential fallback can select the Bedrock default model and then construct an OpenAI client with that Bedrock-only model ID. Please keep provider and model selection consistent when Bedrock cannot resolve AWS credentials, and cover this case as described inline.

def _resolve_default_chat_model() -> str:
"""Return the default chat model for the endpoint that will be used."""
if has_provider_binding():
if provider_is_authoritative(get_metadata_provider()):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This selects the Bedrock model before confirming that Bedrock can create a client. With SKILLSPECTOR_PROVIDER=bedrock, no AWS credentials, and OPENAI_API_KEY set, BedrockProvider.create_chat_model() returns None and create_chat_model_with_provider() falls back to OpenAI using this Bedrock model ID. ChatOpenAI construction can succeed, is_llm_available() can report true, and the first request then targets the wrong endpoint/model combination. Please preserve a provider/model-consistent fallback (or make explicit Bedrock selection fail instead) and add this mixed-credential regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants