Skip to content

fix(grpc): send Struct error details to gRPC clients - #515

Draft
gulldan wants to merge 1 commit into
a2aproject:mainfrom
gulldan:fix/grpc-struct-error-details
Draft

gulldan wants to merge 1 commit into
a2aproject:mainfrom
gulldan:fix/grpc-struct-error-details

Conversation

@gulldan

@gulldan gulldan commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Closes #514. Follow-up to #508.

Problem

a2a_error_to_status encodes the google.rpc.Status through tonic_types::ErrorDetails, which carries only ErrorInfo and BadRequest. Every other TypedDetail on an A2AError (for example TypedDetail::from_struct) is dropped before it reaches a gRPC client, while the JSON-RPC and REST transports carry it in error.data. Since #508 the gRPC client decodes google.protobuf.Struct details, the shape a2a-go sends, so the Rust server was the only side not sending them.

Fix

ErrorDetails has no slot for a custom Any, so the function keeps building ErrorInfo (reason from the code, protocol domain, merged metadata) and BadRequest exactly as before and, only when the error has google.protobuf.Struct details, decodes the google.rpc.Status tonic produced, appends each of them as a Struct Any (errordetails::STRUCT_TYPE, value from a2a_pb::pbconv::to_proto_struct) and re-encodes it with tonic::Status::with_details. Wire order stays ErrorInfo, BadRequest, then the Struct details in the order the error lists them. A status without such details is byte-identical to today's. The public signature is unchanged.

A detail of any other type (say google.rpc.RetryInfo built with TypedDetail::new) has no protobuf encoding here, so it is still dropped as on main, now with a tracing::warn! naming the type URL. The earlier revision sent it as a Struct the way a2a-go does; I took that back because the specification (§3.3.2) requires each detail's @type to identify its type, and a RetryInfo arriving as a Struct does not. Encoding the standard google.rpc types is a separate change (ProtoJSON for error_details.proto in a2a-pb). Numbers in a Struct pass through f64, so integers above 2^53 lose precision; the rustdoc says so.

The status travels base64-encoded in the grpc-status-details-bin trailer, so a large detail turns the error into a transport failure at the client once the header list exceeds what its HTTP/2 stack accepts: measured with GrpcTransport (hyper: 16 KiB), a Struct detail of 11 000 bytes still arrives and from 12 000 bytes the client gets h2 protocol error with no details at all. The error trailers are therefore kept within MAX_ERROR_TRAILER_BYTES = 8 KiB, the metadata size gRPC clients accept by default (grpc-java's maxInboundMetadataSize on Netty, which fails the RPC above 8192; C-core's soft grpc.max_metadata_size, above which RPCs are rejected at random up to the 16 KB hard limit; the limit grpc-go is moving to), measured as HTTP/2 counts a header list (RFC 9113 §6.5.2: name + value + 32 per field). trailer_bytes runs tonic's own Status::into_http on the candidate status, so content-type, grpc-status, the percent-encoded grpc-message and the base64 grpc-status-details-bin are counted as tonic sends them, and adds the two fields tonic does not write: :status (RFC 9113 §8.3.2) and date (RFC 9110 §6.6.1, added by hyper). The message, ErrorInfo and BadRequest always go; the first Struct detail that does not fit and the ones after it are dropped with a tracing::warn!, and the client still receives the error. grpc-go and grpc-java check the peer's advertised SETTINGS_MAX_HEADER_LIST_SIZE when sending; a tonic service cannot see it, so the default is the budget. The TypedDetail rustdoc in a2a now says that gRPC may drop details that do not fit, so an agent author learns it before the warning does.

This limit is a property of gRPC, not of this crate (grpc.io, Error handling: "Larger error detail payloads may run into protocol limits (like max headers size), effectively losing the original error"), and the specification does not mention it; a2a-go sends every detail with no budget and a2a-python sends none but ErrorInfo and BadRequest. I raised that in a2aproject/A2A#2433 with the sources and an SDK table; this PR is a draft until there is an answer on how the bindings should agree (see the comment below).

Tests

  • test_struct_details_round_trip_through_a_grpc_status: a Struct detail survives a2a_error_to_status → status_to_a2a_error unchanged (fails on main: only the ErrorInfo comes back).
  • test_error_info_and_bad_request_stay_first_on_the_wire: the raw google.rpc.Status lists ErrorInfo, BadRequest, Struct in that order even when the error lists the Struct first (fails on main).
  • test_details_of_other_types_are_not_sent_as_structs: a RetryInfo detail does not reach the client as a Struct; only the ErrorInfo does (passes on main, guards the choice above).
  • test_status_without_other_details_is_unchanged: an error with only ErrorInfo and BadRequest produces the same details bytes as tonic::Status::with_error_details (passes on main, guards the existing encoding).
  • test_error_trailers_stay_within_the_default_grpc_metadata_limit: small, over-limit, small; only the first small detail survives and the trailers stay within 8 KiB.
  • test_the_message_counts_against_the_trailer_budget: a long message keeps its place and the detail is dropped instead.
  • test_trailer_bytes_counts_fields_as_http2_does: the accounting matches tonic's encoding of content-type, grpc-status, grpc-message and grpc-status-details-bin.
  • e2e grpc_transport_receives_struct_details_from_a_real_server: a real tonic server returns the Struct detail to GrpcTransport (fails on main).
  • e2e grpc_transport_still_receives_an_error_whose_detail_is_too_large: a 64 KiB detail; the client gets the code, the message and the ErrorInfo (without the cap the client gets an h2 protocol error and nothing else).

CI mirror (ci.yml on Linux): fmt, cargo hack check/clippy --each-feature -D warnings, headers, 977 tests passed. Patch coverage as coverage.yml computes it: 189 added lines, 0 missed.

Overlap

Touches a2a-grpc/src/errors.rs, a2a-grpc/tests/e2e.rs and four doc lines in a2a/src/errordetails.rs. No overlap with the open A2A 1.1 stack. Based on main at 505dbad.

🤖 Generated with Claude Code

@gulldan
gulldan requested a review from a team as a code owner November 6, 2026 12:06
@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.16514% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
a2a-grpc/src/errors.rs 98.16% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@muscariello muscariello left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is the half I left for a follow-up, and it reads well. The wire order is pinned, a status without other details is byte-identical to before, and the round-trip and e2e cases cover the client side from #508.

One thing to check before merging:

Size of the details in the status. The ErrorInfo and BadRequest that went into grpc-status-details-bin were always small. It now carries whatever an agent puts in A2AError::details, and it travels in the response headers or trailers, which HTTP/2 stacks cap (I believe hyper's default is around 16 KB, from memory, not measured). A large detail could turn a clean error into a transport failure at the client, which is worse than dropping the detail. JSON-RPC and REST have no such limit. Could you add an e2e case with a detail of about 64 KB and see what the client receives? If it breaks the stream, cap the encoded size and drop the extra Structs, with a log line.

Smaller, not blocking:

  • The rustdoc of a2a_error_to_status should say that other details are sent as Struct, lose their @type, and pass numbers through f64 (integers above 2^53 lose precision).
  • Status::decode(...).unwrap_or_default() would silently drop the ErrorInfo and BadRequest if decoding ever failed. It can't today, since tonic just produced the bytes, but an expect, or building the Anys directly, would say so.

Otherwise this looks good to me.

@gulldan
gulldan force-pushed the fix/grpc-struct-error-details branch 2 times, most recently from fab0e2a to 7cc9e35 Compare November 6, 2026 15:51
@gulldan

gulldan commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Measured rather than guessed: with a stock GrpcTransport against the test server, a Struct detail of 11 000 bytes still arrives, and from 12 000 bytes the client gets h2 protocol error: http2 error (code INTERNAL_ERROR, no details at all), which matches h2's 16 KiB header list default once the status is base64-encoded into grpc-status-details-bin. So yes, it broke the stream, and the fix is in the amended commit:

  • MAX_STATUS_DETAILS_BYTES = 8 * 1024 caps the encoded google.rpc.Status; the first detail that does not fit and the ones after it are dropped, with one tracing::warn! naming the count and the limit. ErrorInfo and BadRequest are never dropped. 8 KiB leaves about 5 KiB of the 16 KiB for grpc-message and the other trailers.
  • test_struct_details_over_the_size_limit_are_dropped_in_order (small, over-limit, small: only the first small one survives) and the e2e grpc_transport_still_receives_an_error_whose_detail_is_too_large (a 64 KiB detail; the client gets the code, the message and the ErrorInfo). The e2e case fails on the previous head with the h2 error above.
  • Rustdoc of a2a_error_to_status now says the other details go as Struct, lose their @type, pass numbers through f64, and are capped.
  • unwrap_or_default() on the decode is an expect now.

Mirror of ci.yml: 975 tests pass; patch coverage 138 lines, 0 uncovered.

@gulldan
gulldan force-pushed the fix/grpc-struct-error-details branch 2 times, most recently from e3287e9 to 3b43b6e Compare November 6, 2026 17:04
@gulldan

gulldan commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

One more revision on the cap, because the first one was not grounded: it capped the raw google.rpc.Status at 8 KiB, which after base64 is about 11 KiB on the wire, above what a C-core client accepts by default. What the limit actually is, from the sources:

  • HTTP/2 counts a header list as name + value + 32 per field, before HPACK (RFC 9113 §6.5.2).
  • gRPC C-core: grpc.max_metadata_size is a soft limit of 8 KB over the whole batch of headers, with a 16 KB hard limit and random rejection in between (include/grpc/impl/channel_arg_names.h). grpc-java: DEFAULT_MAX_HEADER_LIST_SIZE = 8192 for inbound headers and trailers. grpc-go: 16 MiB today, upcomingDefaultHeaderListSize = 8 << 10. hyper, behind tonic clients: 16 KiB, which is the limit the 64 KiB e2e hit.
  • grpc-go and grpc-java check the peer's advertised SETTINGS_MAX_HEADER_LIST_SIZE on the sending side and fail the RPC with INTERNAL if the trailers exceed it. A tonic service cannot see that setting, so the default every client accepts is the budget.

So the commit now keeps the whole error trailer block within MAX_ERROR_TRAILER_BYTES = 8 KiB, measured the HTTP/2 way: trailer_bytes runs tonic's own Status::add_header on the candidate status (so grpc-status, the percent-encoded grpc-message and the base64 grpc-status-details-bin are counted exactly as sent) and adds the :status, content-type and date fields hyper and tonic put around a trailers-only response. The message, ErrorInfo and BadRequest always go; the first other detail that does not fit and the ones after it are dropped with one warning. test_trailer_bytes_counts_fields_as_http2_does pins the arithmetic against tonic's encoding, test_the_message_counts_against_the_trailer_budget shows a long message takes precedence over a detail, and the 64 KiB e2e is unchanged.

Mirror of ci.yml: 977 tests pass; patch coverage 182 lines, 0 uncovered.

@gulldan

gulldan commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

I have converted this to a draft: re-reading the sources turned up questions I would rather put to you than decide here. The commit is updated (a6ecb51, same single commit, CI mirror 977 tests, patch coverage 189/0), so the current state is visible, but the shape of the fix depends on the answers.

What I found

  1. The specification and a2a-go disagree on @type. §3.3.2 says each detail "MUST include a @type key that identifies the object's type". a2a-go's ToGRPCError (internal/grpcutil/errors.go:99-113) sends every typed detail other than ErrorInfo as a google.protobuf.Struct of its fields, BadRequest included, so a RetryInfo or a BadRequest arrives under @type: google.protobuf.Struct. The previous revision of this PR did the same for a2a-go compatibility. The current commit sends a detail as Struct only when its type is Struct and drops the rest with a warning, as main does, so nothing is sent under a type that is not its own; the client side from fix(grpc): keep status details and the gRPC code in client errors #508 is unchanged.
  2. No two SDKs agree on what travels beyond ErrorInfo (main on 2026-11-06): a2a-go sends everything as Struct and reads ErrorInfo and Struct; a2a-python sends BadRequest and reads ErrorInfo and BadRequest; a2a-java folds the details map into ErrorInfo.metadata; a2a-js and a2a-dotnet send and read ErrorInfo only; a2a-rs sends BadRequest and, since fix(grpc): keep status details and the gRPC code in client errors #508, reads ErrorInfo, BadRequest and Struct. So the BadRequest this server sends is dropped by a Go client, and the Struct this PR adds is dropped by a Python, Java, JS or .NET client.
  3. The size limit is gRPC's and the specification does not mention it. grpc.io's Error handling page: "Larger error detail payloads may run into protocol limits (like max headers size), effectively losing the original error". The clients' defaults are 8192 on grpc-java/Netty (one byte over and Netty rejects the header block), 8 KB soft / 16 KB hard with random rejection in between on C-core, 16 KiB on hyper, and grpc-go is moving from 16 MiB to 8 KiB. a2a-go has no budget, so a large detail there costs the whole error. The budget in this PR is measured on tonic's own Status::into_http output, with :status (RFC 9113 §8.3.2) and date (RFC 9110 §6.6.1) added, and is capped at 8 KiB.

I opened a2aproject/A2A#2433 with the sources and the SDK table, asking the specification to say which details are always sent, that the gRPC binding may drop the rest, and how a type outside google.rpc is encoded.

What I would like to know

  • Should a2a-rs follow a2a-go on the wire (send every non-ErrorInfo detail as a Struct, the previous revision) or the specification's @type rule (the current commit: Struct only for Struct, others dropped with a warning, BadRequest typed as today)? I lean to the current commit, since the Struct form of a BadRequest is read by no client in this organization, but it is your call.
  • Do you want this merged now with the 8 KiB budget, or held until the specification issue has an answer? If held, I can reduce it to the Struct detail round-trip and keep the budget as a separate change.

The rustdoc of TypedDetail in a2a now mentions that gRPC may drop details that do not fit, so an agent author sees it before the warning does; drop it if you prefer a2a not to know about gRPC.

`a2a_error_to_status` encoded only `ErrorInfo` and `BadRequest`, so any other
`TypedDetail` on an `A2AError` (for example `TypedDetail::from_struct`) was
dropped on the way to a gRPC client, while the JSON-RPC and REST transports
carry it. Append every `google.protobuf.Struct` detail to the `google.rpc.Status`
after the `ErrorInfo` and `BadRequest` that `tonic_types::ErrorDetails`
produces; a status with no such detail is encoded exactly as before. A detail
of any other type has no protobuf encoding here and is still dropped, now with
a warning, rather than sent under a type URL that is not its own: the
specification (§3.3.2) requires each detail's `@type` to identify its type.

The status travels base64-encoded in the `grpc-status-details-bin`
trailer, and a large detail turns the error into a transport failure at
the client once the header list exceeds what its HTTP/2 stack accepts
(hyper: 16 KiB). Keep the error trailers within the 8 KiB that gRPC
clients accept by default (grpc-java's `maxInboundMetadataSize` on Netty,
C-core's soft `grpc.max_metadata_size`, the limit grpc-go is moving to),
measured as HTTP/2 counts a header list: the message, `ErrorInfo` and
`BadRequest` come first, and the detail that does not fit and the ones
after it are dropped with a warning.

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
Signed-off-by: gulldan <5641375+gulldan@users.noreply.github.com>
@gulldan
gulldan force-pushed the fix/grpc-struct-error-details branch from a6ecb51 to e998882 Compare November 20, 2026 12:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: gRPC server drops every error detail that is not ErrorInfo or BadRequest

2 participants