fix(providers): treat an env-selected Bedrock provider as an available LLM - #527
arniecommits wants to merge 3 commits into
Conversation
…e LLM
`is_llm_available()` decides whether `create_graph()` wires the analyzers
that set `requires_api_key = True` — the three `semantic_*` ones. It
probes the real chat model only when `has_provider_binding()` is true,
and that reports whether a provider was injected via `use_provider()`,
which only the programmatic/MCP path does. Selecting a provider with
`SKILLSPECTOR_PROVIDER` — the only way from the CLI — leaves that
ContextVar empty, so the check falls through to `_resolve_llm_credentials()`,
which understands API keys and not the boto3 credential chain.
With `SKILLSPECTOR_PROVIDER=bedrock` and working AWS credentials the
result is:
get_active_provider() -> BedrockProvider
has_provider_binding() -> False
is_llm_available() -> (False, "No LLM API key configured...")
so all three semantic analyzers are dropped, while `meta_analyzer` (wired
outside that loop) and `mcp_tool_poisoning` run on Bedrock in the same
process and bill real tokens. The availability check disagrees with the
model path it is checking. `_resolve_default_chat_model()` has the same
shape and falls through to OpenAI for the same reason.
`get_model_config_provider()` already gets this right — it treats an
explicit binding, a CLI provider, or Bedrock as authoritative, and its
docstring says so. This extracts that predicate as
`provider_is_authoritative()` and uses it at the two sites in
`llm_utils.py` that lack it, so the three cannot drift.
Behaviour for other providers is unchanged: CLI providers are still
short-circuited earlier, and an unbound API-key provider is still judged
on its credentials. A Bedrock setup with no working credentials now fails
the model probe instead of the API-key lookup, reporting the provider's
own reason rather than a misleading "no API key".
Verified on a real scan with `SKILLSPECTOR_PROVIDER=bedrock`: before, the
report lists no `semantic_*` entries in
`analysis_completeness.analyzer_statuses`, `llm_available` is false and
`meta_analysis_applied` is false; after, all three run, `llm_calls`
goes 2/2 -> 5/5, `meta_analysis_applied` is true, and the same skill
yields 16 findings instead of 8.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Arnab Roy <alice@cybereducad.com>
`has_provider_binding` is no longer referenced in `llm_utils.py` — both call sites now go through `provider_is_authoritative` — so the import tripped F401. Two of the new tests also needed ruff's formatting. `make lint` and `make format-check` both pass locally now, and tests/unit/test_llm_utils.py and tests/unit/test_bedrock_provider.py are 66 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Arnab Roy <alice@cybereducad.com>
…VIDIA#524 Raise reviewed_pr_through to 527 and reviewed_issue_through to 524 in tools/upstream_baseline.json (commit axis unchanged at 69dcdfb). Every item gets a verdict in docs/DECISIONS.md: NVIDIA#493/NVIDIA#507/NVIDIA#508/NVIDIA#511 verified via git merge-base --is-ancestor as already included through the 2.11.1/2.11.2 sync (including NVIDIA#521, which merged only into the still- open NVIDIA#516 stack, not main); the remaining 27 items stay "wait for upstream merge", none adopted now. Two items get dedicated comparison notes per docs/DIVERGENCE.md's static_runner.py and scripts/compare_scan_accuracy.py rows: NVIDIA#522 uses a different env var name and different default/semantics than this fork's SKILLSPECTOR_MAX_STATIC_SECONDS, so merging it cannot simply delete the divergence row and needs a downstream env var migration first; NVIDIA#490 extends this fork's own upstream PR NVIDIA#486 with a Python 3.14/POSIX edge case the fork's Windows environment does not hit, so NVIDIA#486 is left untouched pending upstream's own resolution. NVIDIA#501-NVIDIA#505 and NVIDIA#518 are also flagged as near-verbatim matches to this fork's existing Windows test divergence rows, worth revisiting for row deletion once merged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: SanHsien <34234698+SanHsien@users.noreply.github.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Reviewed head f84febb08d53985305bf8131e48ae550db6e7d37 — APPROVE.
The change consistently reuses the provider-authority predicate for model selection and availability, preserves OpenAI fallback behavior for unbound API-key providers, and adds focused Bedrock success/failure coverage. I found no required code, test, documentation, or security changes.
Merge gate: required lint, unit, and DCO checks pass, but GitHub currently reports mergeStateStatus=BEHIND; update against current main and re-run required checks before merging.
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Corrective re-review of current head 794edf7de7e5104e0b5b6c1b7a2cad8aa56f79f6. A mixed-credential fallback can select the Bedrock default model and then construct an OpenAI client with that Bedrock-only model ID. Please keep provider and model selection consistent when Bedrock cannot resolve AWS credentials, and cover this case as described inline.
| def _resolve_default_chat_model() -> str: | ||
| """Return the default chat model for the endpoint that will be used.""" | ||
| if has_provider_binding(): | ||
| if provider_is_authoritative(get_metadata_provider()): |
There was a problem hiding this comment.
This selects the Bedrock model before confirming that Bedrock can create a client. With SKILLSPECTOR_PROVIDER=bedrock, no AWS credentials, and OPENAI_API_KEY set, BedrockProvider.create_chat_model() returns None and create_chat_model_with_provider() falls back to OpenAI using this Bedrock model ID. ChatOpenAI construction can succeed, is_llm_available() can report true, and the first request then targets the wrong endpoint/model combination. Please preserve a provider/model-consistent fallback (or make explicit Bedrock selection fail instead) and add this mixed-credential regression.
The problem
With
SKILLSPECTOR_PROVIDER=bedrockand working AWS credentials, the threesemantic_*analyzers are silently dropped from the graph:…while
meta_analyzerandmcp_tool_poisoningrun on Bedrock in the same process and bill real tokens. The scan still exits successfully, so the only sign that detection was halved is a warning on stderr.Root cause
create_graph()drops any analyzer withrequires_api_key = Trueunlessis_llm_available()is true. That function probes the real chat model only whenhas_provider_binding()is true — and that reports whether a provider was injected throughuse_provider(), which only the programmatic/MCP path does. Choosing a provider withSKILLSPECTOR_PROVIDER, the only way available from the CLI, leaves the ContextVar empty, so the check falls through to_resolve_llm_credentials(), which understands API keys and knows nothing about the boto3 credential chain.Reproduced directly:
BedrockProvider.resolve_credentials()returnsNoneby design — its own docstring notes that the OpenAI fallback "is irrelevant for Bedrock" — so no API-key path can ever see it._resolve_default_chat_model()has the same shape and falls through to OpenAI for the same reason.The fix
get_model_config_provider()already gets this right: it treats an explicit binding, a CLI provider, or Bedrock as authoritative, and its docstring says as much. This extracts that predicate asprovider_is_authoritative()and uses it at the two sites inllm_utils.pythat lack it, so the three cannot drift apart.Nothing changes for other providers: CLI providers are still short-circuited earlier, and an unbound API-key provider is still judged on its credentials. A Bedrock setup with no working credentials now fails the model probe rather than the API-key lookup, so the reported reason is the provider's own instead of a misleading "no API key".
Verification
On a real scan of the same skill,
SKILLSPECTOR_PROVIDER=bedrock:semantic_*inanalyzer_statusesmetadata.llm_availablefalsetruemetadata.llm_calls_succeededmetadata.meta_analysis_appliedfalsetrueFive unit tests added in
tests/unit/test_llm_utils.py, covering both directions — that Bedrock is authoritative without an API key, that an unbound API-key provider still is not, and that a Bedrock provider which cannot build a model reports its own error.tests/unit tests/provider tests/nodesrun identically before and after (12 pre-existing environment-dependent failures in both; 3803 → 3808 passing, the five new tests).🤖 Generated with Claude Code