Skip to content

icp-cli: agents use 'candid:' instead of 'didFile:' in the bindgen Vite plugin config #367

Description

@marc0olo

Surfaced while running evals for the #364 PR. Pre-existing — not introduced by that PR (verified below), so filing separately rather than widening its scope.

Symptom

evaluations/icp-cli.json case 15 ("Full-stack Motoko config artifacts") fails one behavior with the skill loaded:

❌ Uses @icp-sdk/bindgen (>= 0.4.0) Vite plugin with a didFile path pointing to the committed .did file
   → The bindgen block uses a 'candid' key instead of the required 'didFile' key,
     and no version is specified for @icp-sdk/bindgen.

The model writes the Vite plugin as icpBindgen({ candid: "...", outDir: "..." }) instead of didFile:. That config is silently wrong — bindgen ignores the unknown key and has no .did path.

Confirmed pre-existing

Case 15 was re-run against the unmodified skill content (whole change set stashed):

BEFORE:  ❌ ... configures the path via a 'candid' key instead of the expected 'didFile' field
AFTER:   ❌ ... uses a 'candid' key instead of the required 'didFile' key

Identical failure, identical reason, 5/6 both ways. The only line the #364 PR touches on that case is the version number inside the expectation string.

Root cause: two different keys for the same concept

The skill uses two key names for "path to the .did file", and the wrong one dominates by weight:

Key Where it belongs Occurrences in skills/icp-cli/
candid: icp.yaml, under recipe.configuration SKILL.md:124, 134, 149, 369, dfx-migration.md:102
didFile: vite.config.js, inside icpBindgen() binding-generation.md:16, 20, dev-server.md:62 — and not once in SKILL.md except one prose aside at :163

An agent reading SKILL.md — the entry point — sees candid: presented four times as the key for the .did path and didFile essentially never. binding-generation.md:91 then compounds it by discussing candid: in the same paragraph as bindgen:

…commit it to the repo, and specify candid: in the recipe config. If candid is omitted, the recipe auto-generates the .did

Nothing anywhere states that these are two distinct keys in two distinct files.

Suggested fix

Make the distinction explicit rather than implicit — one sentence in binding-generation.md next to the Vite example, and a clause in SKILL.md pitfall 12:

The bindgen Vite plugin key is didFile, not candid. candid: is an icp.yaml recipe.configuration key that tells the recipe where the .did is; didFile: tells the binding generator. They point at the same file from two different config files, and bindgen silently ignores an unknown candid key.

Then re-run node scripts/evaluate-skills.js icp-cli --eval 15 — it should go 6/6.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    generatedCreated by an AI agenthallucinationSkill produces incorrect or fabricated outputskill-improvementImprovement to an existing skill

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions