Skip to content

fix(turn): retry requests rejected with 438 (Stale Nonce) - #338

Open
adhami3310 wants to merge 1 commit into
webrtc-rs:masterfrom
adhami3310:fix/turn-stale-nonce-retry
Open

adhami3310 wants to merge 1 commit into
webrtc-rs:masterfrom
adhami3310:fix/turn-stale-nonce-retry

Conversation

@adhami3310

@adhami3310 adhami3310 commented Nov 21, 2026 •

Copy link
Copy Markdown

Fixes #263, fixes #267, fixes #334.

Disclosure: this PR was written with an AI coding assistant (Claude Code), under my direction. I reviewed the change, and every test, number and claim below was run or checked; RFC references were checked against the RFC text. Happy to rework anything.

Problem

A TURN server may retire its nonce at any time and answer the next request on an allocation with 438 (Stale Nonce) and a new nonce. RFC 8489 §9.2.5: "the client MUST retry the request, using the new NONCE attribute supplied in the 438 (Stale Nonce) response." The client stores the new nonce but never retries, and each of the three requests a long-lived allocation keeps sending breaks differently:

Request What happened Issue
Refresh The next refresh is scheduled half a lifetime away, which is when the allocation expires. The server deletes it, and every connection relayed through it fails. #263
ChannelBind The binding is deleted, and the next packet to the peer binds it to a new channel number. A server that still holds the first binding refuses that with 400 for as long as that binding lives (RFC 8656 §12.2: "The transport address is not currently bound to a different channel number"). Meanwhile the ChannelData the server keeps sending on the first number finds no binding at the client. #267
CreatePermission A periodic permission refresh is dropped silently, and the permissions lapse after 5 minutes. #334

Where we hit it: Amazon Chime's TURN servers, which rotate nonces. Every relayed call dropped after ~597 s, logging ~1,800 bind() failed: ChannelBind error response (error 400: ) warnings and ~3,000 handle_read error: no binding found for channel errors along the way.

Fix

Retrying: each of the three requests is sent again at once with the new nonce. The retry is capped by MAX_RETRY_ATTEMPTS (3) consecutive 438s, a constant that already existed but was unused.

What a retry keeps (carried in a crate-private StaleNonceRetry, part of TransactionType; the public API is unchanged and cargo semver-checks reports no semver update required):

  • Refresh: the lifetime it asked for, so a close() (LIFETIME=0) stays a close and isn't turned into a renewal.
  • ChannelBind: its channel number. If the retries run out, the binding is kept (state Idle, same number), so the next packet binds that number again instead of a new one.
  • Results under the caller's id: a retry is a new transaction (RFC 8489 §5: a new transaction ID unless the request is bit-wise identical). But what's reported about it, a CreatePermissionResponse/CreatePermissionError or a TransactionTimeout, names the id the caller was handed. This is the same convention fix(turn): report an Allocate timeout under the id allocate() returned #281 established for Allocate.

Logging:

  • A periodic permission refresh that still fails is now logged with warn! instead of dropped.
  • set_nonce_from_msg's log lines named "refresh allocation" for every request; they now name the actual request.

Tests

Eight new tests in client_test.rs drive the client through its sans-I/O interface: a request goes out, a hand-built 438 with a new NONCE comes back, and the test checks what goes out next.

Test Checks
…_refresh_rejected_for_a_stale_nonce_is_sent_again_with_the_new_nonce new transaction id, new nonce, same lifetime; the refresh schedule resumes on success
…_close_rejected_for_a_stale_nonce_is_sent_again_as_a_close LIFETIME=0 kept; the relay is released
…_request_rejected_for_a_stale_nonce_is_retried_a_bounded_number_of_times 1 + MAX_RETRY_ATTEMPTS sends, then stop
…_permission_rejected_for_a_stale_nonce_is_retried_and_reported_under_its_first_id no error event during the retry; the response is reported under the original id
…_permission_refresh_rejected_for_a_stale_nonce_is_retried the periodic refresh is retried for the same peers
…_channel_bind_rejected_for_a_stale_nonce_keeps_its_channel_number same number on retry, then ChannelData on that number
…_channel_bind_still_rejected_after_its_retries_is_bound_again_on_the_same_number binding kept and rebound on the same number
…_retried_permission_that_is_never_answered_times_out_under_its_first_id the timeout is reported under the original id

Each test except the retry bound fails when the old behaviour is restored: retries disabled and the binding dropped on 438, or the old timeout id, for the timeout test. The existing tests that call handle_refresh_allocation_response directly were updated for its new arguments.

Run locally:

  • cargo test --workspace --no-fail-fast: 2,050 passed.
  • The aws-lc-rs matrix (--no-default-features --features rtc/crypto-aws-lc-rs): 1,694 passed.
  • cargo test --workspace --doc: 344 passed.
  • cargo semver-checks --workspace --default-features: no update required.
  • rustfmt --check clean on the changed files.
  • cargo clippy -p rtc-turn --all-targets adds no new warnings; the 7 in rtc-turn/src predate this.

Live, against Amazon Chime's TURN servers, with the same change backported to rtc-turn 0.21.1, the version webrtc 0.21.1 uses:

  • A 16-minute relayed session ran with no ICE failure, no reconnect, and none of the 400 / "no binding found" errors. Before the change, sessions failed at ~597 s.
  • An earlier version of this change, which retried after one second instead of at once, also held a 15-minute two-party call with no drop.
  • Caveat: the TURN client's debug output was filtered in the 16-minute run, so its log doesn't show how many 438s arrived during it.

Not in this PR

A TURN server may retire the nonce it issued at any time and answer the
next request on the allocation with 438 and a new one. RFC 8489 §9.2.5
has the client retry the request with that nonce. The client stored the
new nonce but did not retry, and each of the three requests a long-lived
allocation keeps sending went wrong in its own way:

- Refresh (webrtc-rs#263): the next refresh is scheduled half a lifetime away,
  which is when the allocation expires. The server deleted it, and every
  connection relayed through it failed.
- ChannelBind (webrtc-rs#267): the binding was deleted, and the next packet to the
  peer bound it to a new channel number. A server that still held the
  first binding refuses that with 400 (RFC 8656 §12.2) for as long as the
  first binding lives, and the ChannelData it kept sending on the first
  number found no binding at the client.
- CreatePermission (webrtc-rs#334): a periodic refresh was dropped silently, and
  the permissions lapsed after 5 minutes.

Each is now sent again at once with the new nonce, up to
MAX_RETRY_ATTEMPTS (3) times in a row; the constant existed but was
unused. A retried Refresh keeps the lifetime it asked for, so a close()
stays a close. A ChannelBind keeps its channel number, and when its
retries run out the binding stays and is bound again on the same number
by the next packet. A retried request is a new transaction, but what is
reported about it, a CreatePermission result or a transaction timeout,
names the id the caller was handed, as an Allocate's does. A periodic
permission refresh that still fails is logged rather than dropped.

Seen against Amazon Chime's TURN servers, which rotate nonces: every
relayed call dropped after ~597 s, with ~1800 "ChannelBind error
response (error 400)" warnings and ~3000 "no binding found for channel"
errors along the way. With an earlier version of this change applied to
0.21.1 (it retried after one second rather than at once), a 15-minute
call ran with no drop and none of those errors.

Fixes webrtc-rs#263, fixes webrtc-rs#267, fixes webrtc-rs#334.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment