Agent Card claims vs. agent behavior: findings from 554 public agents

[Kristjanh2]

We've been testing A2A agents against their own Agent Cards and wanted to share the numbers, since they say a lot about where the ecosystem is.

From the A2A Registry public API (554 agents, Oct 13):

• 97% of cards pass the registry's conformance check
• 98% pass the health check, but only 45% (248) return a working answer to a basic task
• 63% still declare protocol 0.3
• 1 in 5 cards declares skills without a single example input

In short: a valid, reachable card says little about whether the agent does what it claims. When your agent starts calling agents built by other companies, the card is all you have to go on, and nothing checks it.

That's the gap we're working on with Suncly: it tests an agent against its own card in a sandbox and produces a signed report per skill that anyone can verify offline. We're extending the same approach to MCP tool annotations.

Question for people running A2A in production: how do you decide today whether to trust an agent you didn't build? Manual review, allowlists, something else?

suncly.com

[capitaharlock]

The 45% working answer vs 98% health check gap is why we don't decide trust once by allowlist or manual review.

We went a different route with MeshKore, treating every card as unverified and re-probing each advertised skill continuously, so trust is earned per skill from live calls; identity is self-asserted, so there's no allowlist to lean on (https://meshkore.com/oracle).

A signed per-skill report is the right direction, but I'd trust the re-check more than the signature.

[Kristjanh2]

Thanks, that’s a fair point, and it matches what we saw in the data: a one-time check goes stale fast.

I see the re-check and the signature as answering different questions. A live probe tells you a skill responds right now. A signed report records what was checked and against which card hash, so it can be verified later without re-running the agent. Scheduled re-runs are on our roadmap too, so the two end up complementing each other.

How does your probe decide that a skill is working? Happy to compare notes.

[imokokok]

The 45% number is the useful one here — it says the failure mode that matters isn't "unreachable"
but "answers badly", and that's exactly the part a health check is structurally unable to see.

On @capitaharlock's point — "I'd trust the re-check more than the signature" — I'd argue
re-checks and signatures aren't competing. They operate on different halves of the problem:

• A re-check tells you how the agent behaves now, at the probe. It's valuable, but notice
  what it does to trust: the oracle becomes a new self-asserted authority, just held by the
  prober instead of the agent author. "Trust MeshKore's live probes" has the same shape as
  "trust the card" — one more party whose claim isn't verifiable by you.
• A signature does something a probe can't: it freezes a specific check — of a specific skill,
  against specific inputs — into a portable artifact that a third party can re-verify offline,
  months later, in a dispute. Without it, "we re-probe continuously" is a claim about your
  operations that evaporates exactly when it's needed: after something goes wrong.

There's also a capability limit worth naming: sandbox re-probing is good at catching agents that
don't respond or crash. It is much weaker at catching the failures that cost money — an agent
whose card says "multi-source price aggregation" but whose pipeline is single-source, or declares
freshness it doesn't have. The prober doesn't know what the answer should be, only whether
there is one. For that gap, a report has to carry what was measured against which committed
inputs — recomputable, not just repeated — which is the measured/asserted split we pushed on in
#1734.

To suncly's actual question, from running a pre-trade risk service that agents call before they
pay: we don't decide trust once, and we don't rely on continuous probing either. Allowlist to
start, a signed risk verdict bound into each payment authorization as the gate, and the receipt
is what a dispute re-opens. Trust isn't decided; it's evidenced per authorization — the probe
and the signature both feed that, but neither substitutes for it.

[Kristjanh2]

Thanks, “evidenced per authorization” is a sharper framing than ours, and I agree the re-check and the signature aren’t competing. One tells you how the agent behaves now, the other gives you something that still holds up later, when someone actually has to check what happened.

The limit you named is real. A check that only knows whether an answer came back can’t catch a card that claims multi-source when the pipeline is single-source. Our rule is that anything a test can’t expose stays unverified and is listed that way in the report, never passed. The measured vs asserted split you described is the right line to draw.

Curious about your side: when you bind a signed risk verdict into a payment authorization, what does it commit to? The agent’s identity, the card hash, the specific inputs?

[imokokok]

@Kristjanh2 — in the combined Insight + PriorSeal path, the signed PriorSeal authorization commits to the action, not to a generic risk label. For an exact EVM call that is the chain, authorized executor, nonce, call target, calldata hash, native value, validity window, and finality constraints; the authorization also binds the principal/authority terms. A transfer authorization instead binds its asset, amount, sender, recipient, nonce, and expiry.

A signed Insight risk assessment can be carried as a namespaced SHA-256 or Keccak-256 contextCommitment to the complete external assessment/decision artifact. Its signature, issuer key, policy, and freshness remain separately verified; the commitment establishes that this exact digest was included in the signed authorization, not that PriorSeal made or independently validated the risk verdict.

There is no generic card-hash field. If a payment-specific artifact needs binding, it should be a namespaced digest of a properly scoped, separately protected artifact—not raw card data—and its semantics and verifier remain external to PriorSeal.

[Kristjanh2]

Thanks, that’s a precise answer, and the separation makes sense: the authorization commits to the action, and anything external rides along as a digest whose meaning and verifier stay outside PriorSeal.

That model looks like a natural fit for an agent attestation. A Suncly report is already a separately signed artifact with its own issuer key and its own offline verifier, so it could be bound as a namespaced contextCommitment without PriorSeal having to interpret anything about the agent or its card. Agreed that raw card data doesn’t belong in the authorization. The report is the properly scoped artifact, and the card hash stays inside it.

Is there a convention for contextCommitment namespaces, or would we propose one? Happy to open an issue on PriorSeal to work it out.

[imokokok]

Thanks — that scope is right. Please tag @imokokok when the issue is open; we’ll review the profile and verifier claim boundary only. No core interpretation, integration, or pilot follows from that review.

[kaspermuutra-dotcom]

@imokokok one small clarification, and a separate offer that has nothing to do with the review. Kristjan (Kristjanh2) and I both work on Suncly, which is why two accounts have replied here.

Separately: Insight is something other agents call before they pay, so if you ever want its agent or MCP endpoint tested against what it declares, we're running early tests at 99% off with code SUNCLY-9BRNWLTC. It's charged only when the report is delivered. You can apply here and we'll reply within 2 hours: https://www.suncly.com/early-testing

No connection to the profile review, and no need to reply if it's not useful.

Kasper

[imokokok]

@Kristjanh2 — there is no central namespace registry today. A PriorSeal context commitment is the sorted triple {namespace, algorithm, digest}: core validates its bounded identifier and digest, but deliberately does not interpret the external artifact. Existing namespaces are versioned artifact identifiers, for example insight.pretrade-pair.v1 and priorseal.erc4337.user-operation.v1.

For a Suncly report, suncly.agent-attestation.v1 would be a sensible starting point if Suncly owns the schema. It should commit to the canonical bytes of the complete signed report—not a separate raw card hash—and the profile must state the canonicalization, digest algorithm, version-upgrade rule, issuer-key/verifier path, and the exact claim that a verifier checks.

Please open a PriorSeal issue for that optional artifact profile. It should not imply core interpretation, an integration commitment, or a pilot; it gives us a focused place to pin those interoperability details before anyone relies on the namespace.

[kaspermuutra-dotcom]

@imokokok thanks, that's exactly the guidance we needed. We'll open the PriorSeal issue for an optional suncly.agent-attestation.v1 artifact profile this week, covering canonicalization of the complete signed report bytes, digest algorithm, version-upgrade rule, issuer-key/verifier path, and the exact claim a verifier checks. Scoped as you described: no core interpretation, no integration or pilot implied.

Kristjan

[kinlane]

A second population for the same numbers, in case it helps separate what's true of the ecosystem from what's true of one registry.

APIs.io fetches /.well-known/agent-card.json (and the legacy agent.json) on every host the companies in its catalog declare, then grades each card against A2A 1.0's structure. These are companies' own hosts, not registry submissions. In the October 30 build:

• 592 providers serve a card, and 466 (79%) pass every structural check. That's well below the 97% registry-conformance figure above. Registry submission seems to select for cards that already validate, while cards found on company domains include near-misses: capabilities sent as an array, interface field mix-ups, no protocolVersion.
• Protocol version: about 67% declare 0.3.x, consistent with your 63%. 14% declare 1.0 and 17% declare none.
• 40 still serve only the legacy /.well-known/agent.json.

So both populations agree that most live cards are on 0.3. Where they differ, on structural conformance, a card found in the wild is less likely to be clean than one submitted to a registry. Behavior is the harder question, as you say. We grade structure only, and no task was sent to any agent.

Per-provider grades: https://apis.io/a2a/. Adopters by grade: https://apis.io/standards/agent2agent/.