Repository navigation
fix(turn): retry requests rejected with 438 (Stale Nonce) - #338
Open
adhami3310 wants to merge 1 commit into
Open
adhami3310 wants to merge 1 commit into
adhami3310 wants to merge 1 commit into
Conversation
A TURN server may retire the nonce it issued at any time and answer the next request on the allocation with 438 and a new one. RFC 8489 §9.2.5 has the client retry the request with that nonce. The client stored the new nonce but did not retry, and each of the three requests a long-lived allocation keeps sending went wrong in its own way: - Refresh (webrtc-rs#263): the next refresh is scheduled half a lifetime away, which is when the allocation expires. The server deleted it, and every connection relayed through it failed. - ChannelBind (webrtc-rs#267): the binding was deleted, and the next packet to the peer bound it to a new channel number. A server that still held the first binding refuses that with 400 (RFC 8656 §12.2) for as long as the first binding lives, and the ChannelData it kept sending on the first number found no binding at the client. - CreatePermission (webrtc-rs#334): a periodic refresh was dropped silently, and the permissions lapsed after 5 minutes. Each is now sent again at once with the new nonce, up to MAX_RETRY_ATTEMPTS (3) times in a row; the constant existed but was unused. A retried Refresh keeps the lifetime it asked for, so a close() stays a close. A ChannelBind keeps its channel number, and when its retries run out the binding stays and is bound again on the same number by the next packet. A retried request is a new transaction, but what is reported about it, a CreatePermission result or a transaction timeout, names the id the caller was handed, as an Allocate's does. A periodic permission refresh that still fails is logged rather than dropped. Seen against Amazon Chime's TURN servers, which rotate nonces: every relayed call dropped after ~597 s, with ~1800 "ChannelBind error response (error 400)" warnings and ~3000 "no binding found for channel" errors along the way. With an earlier version of this change applied to 0.21.1 (it retried after one second rather than at once), a 15-minute call ran with no drop and none of those errors. Fixes webrtc-rs#263, fixes webrtc-rs#267, fixes webrtc-rs#334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #263, fixes #267, fixes #334.
Problem
A TURN server may retire its nonce at any time and answer the next request on an allocation with 438 (Stale Nonce) and a new nonce. RFC 8489 §9.2.5: "the client MUST retry the request, using the new NONCE attribute supplied in the 438 (Stale Nonce) response." The client stores the new nonce but never retries, and each of the three requests a long-lived allocation keeps sending breaks differently:
Where we hit it: Amazon Chime's TURN servers, which rotate nonces. Every relayed call dropped after ~597 s, logging ~1,800
bind() failed: ChannelBind error response (error 400: )warnings and ~3,000handle_read error: no binding found for channelerrors along the way.Fix
Retrying: each of the three requests is sent again at once with the new nonce. The retry is capped by
MAX_RETRY_ATTEMPTS(3) consecutive 438s, a constant that already existed but was unused.What a retry keeps (carried in a crate-private
StaleNonceRetry, part ofTransactionType; the public API is unchanged andcargo semver-checksreports no semver update required):close()(LIFETIME=0) stays a close and isn't turned into a renewal.Idle, same number), so the next packet binds that number again instead of a new one.CreatePermissionResponse/CreatePermissionErroror aTransactionTimeout, names the id the caller was handed. This is the same convention fix(turn): report an Allocate timeout under the id allocate() returned #281 established for Allocate.Logging:
warn!instead of dropped.set_nonce_from_msg's log lines named "refresh allocation" for every request; they now name the actual request.Tests
Eight new tests in
client_test.rsdrive the client through its sans-I/O interface: a request goes out, a hand-built 438 with a newNONCEcomes back, and the test checks what goes out next.…_refresh_rejected_for_a_stale_nonce_is_sent_again_with_the_new_nonce…_close_rejected_for_a_stale_nonce_is_sent_again_as_a_closeLIFETIME=0kept; the relay is released…_request_rejected_for_a_stale_nonce_is_retried_a_bounded_number_of_timesMAX_RETRY_ATTEMPTSsends, then stop…_permission_rejected_for_a_stale_nonce_is_retried_and_reported_under_its_first_id…_permission_refresh_rejected_for_a_stale_nonce_is_retried…_channel_bind_rejected_for_a_stale_nonce_keeps_its_channel_number…_channel_bind_still_rejected_after_its_retries_is_bound_again_on_the_same_number…_retried_permission_that_is_never_answered_times_out_under_its_first_idEach test except the retry bound fails when the old behaviour is restored: retries disabled and the binding dropped on 438, or the old timeout id, for the timeout test. The existing tests that call
handle_refresh_allocation_responsedirectly were updated for its new arguments.Run locally:
cargo test --workspace --no-fail-fast: 2,050 passed.--no-default-features --features rtc/crypto-aws-lc-rs): 1,694 passed.cargo test --workspace --doc: 344 passed.cargo semver-checks --workspace --default-features: no update required.rustfmt --checkclean on the changed files.cargo clippy -p rtc-turn --all-targetsadds no new warnings; the 7 inrtc-turn/srcpredate this.Live, against Amazon Chime's TURN servers, with the same change backported to
rtc-turn0.21.1, the versionwebrtc0.21.1 uses:Not in this PR
webrtc-rs/webrtchas its ownturncrate. It returnsErrTryAgainon 438 the way Pion does, and I haven't checked whether it has these problems.CHANGELOG.mdhas one entry under Unreleased → Fixed.