Skip to content

fix: keep runtime-selected printf reconstruction incomplete - #514

Open
mohgupta-ship-it wants to merge 34 commits into
mainfrom
codex/fix-runtime-command-completeness
Open

mohgupta-ship-it wants to merge 34 commits into
mainfrom
codex/fix-runtime-command-completeness

Conversation

@mohgupta-ship-it

@mohgupta-ship-it mohgupta-ship-it commented Sep 10, 2026

Copy link
Copy Markdown
Member

Runtime-selected executable and wrapper names can fall out of bounded printf reconstruction and incorrectly produce a complete/SAFE result. Recognize ambiguous invocations from printf-style operands or destructive outer arguments and report partial analysis so strict CLI and MCP installation gates reject unresolved work.

Preserve ordinary parameter documentation, PowerShell expressions, and the reference-accounting fixes from #507. Operand lookahead remains bounded. Clone monitoring tolerates disappearing Git metadata only while the clone is active and still performs a strict final inspection.

Regression tests distinguish literal shell backticks from Markdown delimiters, including longer code spans that contain actual shell substitutions. Paired CLI/MCP tests exercise static-only and semantic-enabled workflows with deterministic model responses and verify that successful LLM analysis does not erase incomplete static coverage. The mocked LLM-failure integration test builds its graph after configuring provider availability.

Validation:

  • Python 3.12, locked dependencies: make test-ci passed 4,179 tests, 14 skipped, 38 deselected, 4 expected failures; 89% coverage.
  • 116 reconstruction contract cases and 8 paired CLI/MCP workflow cases passed.
  • Offline graph integration, Ruff lint/format, diff checks, DCO, and secret scan passed.
  • Production source and Docker inputs are unchanged from the previously validated PR head; changes in this follow-up are tests only.

The full suite was rerun under controlled load after concurrent suites caused two large-file timing failures; both cases passed without changing their assertions or resource limits. Joint validation with the documentation and quote-scan fixes is tracked separately.

Prepared by Codex on behalf of Mohit Gupta.

Prepared by Codex for Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Allow missing Git metadata only while cloning is active, then require a strict final measurement. Preserve permission failures, checkout errors, and all ingest limits. Cover disappearing files and directories, final budget enforcement, and fail-closed controls.

Prepared by Codex for Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Build the integration-test graph after configuring mock availability so semantic transports are exercised without provider credentials.

Prepared by Codex for Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Run CLI and MCP gates through static-only and semantic-enabled workflows using deterministic model responses. Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked these seven cases against this commit and the base. The PowerShell issue is new; the other six are existing gaps that still show up here.

Comment thread tests/nodes/analyzers/test_security_reconstruction.py
Comment thread src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py
Comment thread src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py Outdated
Comment thread tests/nodes/test_security_end_to_end.py
Comment thread src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py
Comment thread src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py
Comment thread src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py
@yashrajp22
yashrajp22 self-requested a review September 11, 2026 07:35

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Reviewed current head 11f15ab947d8db05c6fb3d8f371eddf946519a28 — REQUEST_CHANGES.

The seven unresolved, non-outdated review threads are reproducible gaps in the fail-closed contract and remain unaddressed on the merge-only head:

  • tests/nodes/analyzers/test_security_reconstruction.py:1721: a runtime-selected destructive command inside a triple-backtick Markdown fence is skipped and can yield complete/SAFE. Route fenced code bodies through the incomplete-analysis check and add this regression.
  • src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:729: assignment/exec/separator prefixes can make the parser stop before $CMD (for example $(MODE=x $CMD %s r m) -rf /). Continue to the runtime command or fail closed.
  • src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:1203: brace-expanded absolute targets such as {/,/tmp} bypass the root-path check. Expand/prove the bounded brace alternatives or retain partial status.
  • tests/nodes/test_security_end_to_end.py:1021: declared-marker reconstruction scans findings but does not propagate parse-exhaustion status, allowing a reconstructed runtime command to become complete/SAFE. Carry the exhaustion result into the ledger.
  • src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:1180: an empty quoted prefix ("""$($CMD)" -rf /) causes tokenizer uncertainty that is discarded. Preserve the tokenizer's limited flag.
  • src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:1186: a destructive target just beyond the 8,192-character lookahead is treated as clean. Reaching the lookahead boundary must produce partial analysis.
  • src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:742: ordinary PowerShell replacement expressions containing %TEMP%, %s, or printf are misclassified as printf reconstruction and blocked. Exclude PowerShell value expressions while retaining unresolved shell-command cases.

The head changed during review from d95663d9766ae214a6b6668dac8bda62b41f5c1e to this merge commit; the current threads are still unresolved. The required fixes, unresolved threads, absent checks on the new head, and mergeStateStatus=BLOCKED all block merging.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed current head 2c6a19c99ed8f46cc0ad5e7aa532c7b50b5abe6d.

The Markdown-fence finding from the previous review is resolved by the synchronized #516 changes: fence delimiters are now masked while fenced command bodies still reach the bounded parse-exhaustion check, with Markdown regression coverage.

The other six prior blockers remain on this head: assignment, exec, or separator prefixes can stop before the runtime command; brace-expanded absolute targets are not expanded by the root-path check; declared-marker projections still do not propagate parse exhaustion; tokenizer uncertainty from an empty quoted prefix is discarded; reaching the 8,192-character lookahead boundary is treated as clean; and ordinary PowerShell replacement strings containing %TEMP%, %s, or printf still trigger the printf-reconstruction heuristic. The existing inline threads contain the exact reproductions and requested fixes, so I have not duplicated them.

No hosted checks are reported on this head, and GitHub reports mergeStateStatus=BLOCKED.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed current head b4971cafc36d895ae265d80d576175c4f7a47d56. The commits since the marked review at 2c6a19c99ed8f46cc0ad5e7aa532c7b50b5abe6d only merge #522's configurable static-analysis allowance. That synchronization changes budget configuration, documentation, and tests but does not alter the six remaining parser/completeness blockers.

Assignment, exec, or separator prefixes can still stop before the runtime command; brace-expanded absolute targets remain unexpanded by the root-path check; declared-marker projections still fail to propagate parse exhaustion; tokenizer uncertainty from an empty quoted prefix is discarded; reaching the 8,192-character lookahead boundary is treated as clean; and ordinary PowerShell replacement strings containing %TEMP%, %s, or printf still trigger the printf-reconstruction heuristic. The existing inline threads contain the exact cases and requested fixes, so I have not duplicated them. The Markdown-fence finding remains resolved by the earlier #516 synchronization.

No hosted checks are reported, and GitHub reports mergeStateStatus=BLOCKED.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed current head 0ad0b8e65a002152b8cfbd3054ab3c9b1c2e0464 after the marked review at b4971cafc36d895ae265d80d576175c4f7a47d56. All six PR-owned production and test blobs are byte-for-byte unchanged.

The Markdown-fence bypass remains resolved by the synchronized #516 work. The other six blockers remain: assignment/exec/separator prefixes can stop before the runtime command; brace-expanded absolute paths evade root-target recognition; declared-marker scans do not propagate parse exhaustion; empty quoted prefixes discard tokenizer uncertainty; reaching the bounded lookahead is treated as clean; and ordinary PowerShell replacement expressions are still mistaken for printf reconstruction. The existing threads contain the exact reproductions and expected corrections, so I have not duplicated them.

No required checks are reported on this head, active change requests and unresolved threads remain, and GitHub reports mergeStateStatus=BLOCKED.

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the time-budget issue below before merging.

Comment thread src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py Outdated

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed current head 4b2584c721179b70443aa1f12739cec1a1cd5f4a. The head change only synchronizes unrelated main work; the PR-owned analyzer and regression files are unchanged. The eight previously reported fail-open and runtime-bound cases are now represented by exact regressions and their threads are resolved. A new regression in the current implementation blocks approval.

At src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:3011-3024, the additive raw-Markdown pass sends the entire unmasked document through _has_shell_command_word_exhaustion. That discards established inline-code, JSON-string, table-cell, and paragraph ownership and marks benign documentation such as $(hostname).example partial. The latest tested implementation failed 172 tests across those contracts and CLI/MCP gates; three parser stress cases also exceeded their 12-second bound. Restrict the raw fallback to proven runtime-selected destructive or backtick evidence (or preserve ownership metadata), restore the completeness contracts, and obtain a green exact-head rerun.

The current head has no attached checks. The correctness/performance regression, missing exact-head validation, active change request, and GitHub BLOCKED state prevent merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants