Skip to content

Quarantine intermittent MARS split-packet test - #5065

Open
mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/flaky-test-quarantine-nov13
Open

mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/flaky-test-quarantine-nov13

Conversation

@mdaigle

@mdaigle mdaigle commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🤖 This pull request was created by a fully automated workflow.

Description

Add [Trait("category", "flaky")] and an observed-error comment to this one fact:

Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSSplitTest

It intermittently times out consuming the pre-login handshake acknowledgement through its test-owned TCP proxy. The exact fact failed and passed on the same upstream-main revision in the same job. Method-level quarantine affects this fact on both frameworks; all bodies, existing categories, four ordinary siblings and pipelines remain unchanged.

Observation Result Build / revision Observed UTC Configuration
Failed 3710387/101801 191699 / 45aacd87d3b9 2026-11-13 01:30:27.740–01:30:43.283 Win25_Sql25 job, net10.0, managed SNI, x64, Release, Project
Passed retry 3710387/102663 Same 2026-11-13 01:38:22.427–01:38:23.160 Identical job/configuration

The unchanged-job retry supplies the comparable passing control. This is quarantine, not a root-cause repair or stability claim. Complete collapsed evidence record includes the image-naming caveat and exact validation commands/results.

Issues

Related: #5047 already quarantined the distinct FullHeaderTest and is untouched; #5038 is a separate pending certificate proposal. #4980 and #4996 are merged discovery inputs; #5039 is separate fixes/probation work. No issue is closed.

Testing

Built and verified actual xUnit/VSTest selection on net10.0 and net462: quarantine selects this fact plus the existing FullHeader control; ordinary selection excludes both and selects four siblings. git diff --check passed.

Local execution was not passing: net10.0 skipped because configuration predicates threw TargetInvocationException; net462 executed and failed with a pre-login timeout. Neither counts as a pass. Three CI run inventories (9,024 results) were reconciled; broader 14-day per-test review remains incomplete following a build-wide MCP transport failure. Discovery checkpoint not advanced.

  • Tests added or updated - category and observed-error comment only
  • Public API changes documented - no API changes
  • Verified against customer repro (if applicable) - CI-only quarantine
  • Ensure no breaking changes introduced - no product or test-body changes

Guidelines

Please review the contribution guidelines before submitting a pull request:

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle requested a review from a team as a code owner November 13, 2026 14:17
Copilot AI balanced review requested due to automatic review settings November 13, 2026 14:17
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 13, 2026
@mdaigle mdaigle added the Area\Tests Issues that are targeted to tests or test projects label Nov 13, 2026
@mdaigle mdaigle added this to the 8.1.0-preview1 milestone Nov 13, 2026
@mdaigle

mdaigle commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author
Quarantine evidence, validation and incomplete discovery checkpoint - 2026-11-13

Supported category-only change

Exact fact, without parameters:

Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSSplitTest

Commit eda177d97 adds only [Trait("category", "flaky")] and an immediately preceding
comment describing the verified SqlException pre-login handshake timeout.
The existing Set 3, conditional predicates, all test bodies, and all other categories
are unchanged. This method-level trait affects this one fact on both net462 and
net10.0; it does not quarantine the class or its other four ordinary facts.
FullHeaderTest was already quarantined by #5047 and is untouched.
No repair, probation transition, pipeline change, or stability claim is included.

Decisive failure and passing control

Both observations are actual, individually verified results for the exact fact.
Both have empty subResults; neither is an inferred grouped-parent outcome.

Observation Result Stored observed UTC start - completion
Failed 191699 / 3710387 / 101801 2026-11-13T01:30:27.740Z - 2026-11-13T01:30:43.283Z
Passed, same ordinary task's retry 191699 / 3710387 / 102663 2026-11-13T01:38:22.427Z - 2026-11-13T01:38:23.160Z

Build 191699 / 31725, definition CI-SqlClient (1879), branch refs/heads/main,
commit 45aacd87d3b9ebd01dce23e6d7f6d9dc9a8fe582.
The run's verified pipeline reference is
Win25_Sql25 / net10_0_AnyCPU_ManagedSNI_3 / __default, stage/phase/job attempt 1.
Configuration: Windows, local/default-instance SQL image ADO-MMS25-SQL25 version
40.0.0, .NET 10, managed SNI, x64, Release, Project references, Set 3,
ordinary manual-test task. This is the local-SQL configuration, not an
enclave-selected stage. No differing configuration or later revision is needed for
the decisive comparison.

Image naming caveat: the nominal Win25_Sql25 stage and pool image must not be
mistaken for independently verified Windows Server 2025. The mapped initialization
log actually reports Windows Server 2022, 10.0.20348, runner image
windows-2022 / 20261101.353.1. Both observations used the same job allocation.
The SQL version is described by the configured SQL25 image; no separate server
version query was performed.

The actual error is Microsoft.Data.SqlClient.SqlException: Connection Timeout Expired
while consuming the pre-login handshake acknowledgement, with
[Pre-Login] initialization=2; handshake=15027 and inner
System.ComponentModel.Win32Exception: The wait operation timed out.
The source comment states that symptom, not a proposed root cause.

Unique timeline mapping, with parent links and attempt checked:

  • Stage b30ceaf8-5b92-587d-23b9-301e8c14ed73 ->
    phase 76a360f0-a9e3-5a52-0455-858b3651dc0c ->
    job 0fd8138c-e14e-5436-a4a5-ae29d00dfe20.
  • Ordinary task/log 4019:
    task e79a0098-2bc7-5256-9ba8-047187bcf8b8, Run Manual Tests x64.
    Authenticated MCP log pages 1-180 and 181-480 show the matching configuration,
    this failure, the retry at 01:33:02.645Z, and the retry's successful summary:
    713 passed, 81 skipped, zero failed at 01:40:38.848Z.
    The first attempt had 712 passed, 81 skipped, one failed.
  • Publish task/log 4107:
    task ed334ed5-2e3d-5616-bcee-1e55a8ad005c explicitly publishes all
    5,951 results to run 3710387.
  • Quarantine task/log 4100:
    Run Flaky Manual Tests x64 starts only at 01:40:39.260Z, after both
    decisive observations. Its command uses TestFilters=category=flaky and
    TestCodeCoverage=false. The ordinary default excludes flaky.
  • Initialization task/log 282
    supplies the actual OS/image observations above.

The first ordinary attempt's only failure was this test; its four ordinary
SplitPacket siblings passed. The entire ordinary retry passed without a revision,
job, SNI, reference-mode, or SQL configuration change. Setup tasks succeeded.
This rules out a deterministic configuration-only explanation and supplies no
evidence of a shared SQL outage in that job. It does not identify or repair the
underlying cause. The same-job exact-result retry, not overall job success, is
the reason quarantine is supported.

Supplemental observations and configuration limits

  • The exact fact also failed in
    191931 / 3713578 / 101023,
    2026-11-13T13:27:59.117Z - 13:28:14.647Z, with the same pre-login timeout
    (handshake=15014). Revision 114f7248f9108d13f005760bdcd9995489c164c0,
    PR Fix async cancellation failing to send TDS attention signal #4435 merge, sqlclient-pr (2281), Windows/.NET 10/managed SNI/Debug,
    ordinary Sets 123. Unique job
    c2f5ecce-247c-5f99-2bff-fd2b4835ddd4, identifier
    test_windows_net10.test_windows_net10_localhost_managedSni_123_sqlclient_manual.__default,
    attempt 1; task e98e8dae-6f0d-5597-7a2c-13d010199322, log 577.
    Its differently configured .NET Framework/Linux passes are not the
    decisive controls. Fix async cancellation failing to send TDS attention signal #4435 changes MARS code, so that build alone cannot exclude
    a branch-specific regression. Upstream-main evidence above avoids that problem.
  • A successful-build observation was individually verified:
    191916 / 3713346 / 100198,
    2026-11-13T12:13:20.390Z - 12:13:21.913Z, commit
    bf366306f0a8210f1289429b2370ccdea7ac239b, PR Connection encryption info – Slice D: Native SNI interop #4836 merge,
    sqlclient-pr (2281), Windows/.NET 10/native SNI/Debug, ordinary Sets 123.
    Unique job 39ac3cda-36de-561c-cc0f-a8f019ddeca6, identifier
    test_windows_net10.test_windows_net10_localhost_nativeSni_123_sqlclient_manual.__default,
    attempt 1; task 02a39e29-5b33-556b-8b5b-cc612aa43344, log 574.
    Initial display-name matching failed; exact identifiers and stage/phase/job
    parent links resolved the mapping. This different revision/SNI/configuration is
    contextual evidence only, not the proof of intermittency.

Enumerated interval, reconciliation and unfinished work

No reliable fully processed tagging checkpoint was found in the prior tagging
records, including #4980, #4996, #5006, #5016, #5020, #5024, #5028, #5038 and
#5047. Fix/probation PR #5039 is context only and cannot advance discovery.
The full completed-build finish-time window was therefore rescanned:
2026-10-30T14:01:46.349Z through 2026-11-13T14:01:46.349Z.

Authenticated MCP rechecked all requested definition names/IDs and Enabled
status. Disabled stress definition 2250 was checked and excluded.
All 520 unique completed builds were enumerated in six bounded pages
(50, 100, 100, 100, 100, 70), including all result states and branches.
No repository-filter pagination shortcut was used.

Definition Failed Partial success Succeeded Canceled Total
CI-SqlClient (1879) 68 49 0 7 124
CI-SqlClient-Package (1917) 30 30 0 4 64
PR-SqlClient-Project (2197) 41 84 0 2 127
PR-SqlClient-Package (2198) 5 11 0 0 16
sqlclient-pr (2281) 47 0 112 2 161
sqlclient-ci-package (2300) 0 0 28 0 28
sqlclient-ci-unit (2309) 0 0 0 0 0

Build inventory complete; whole-window per-test review incomplete.

Authenticated build-result discovery covered builds 191931, 191888, 191699,
191916 and canceled 191848
for adverse outcomes. Passed discovery completed
for 191931, 191888 and 191916; the build-wide Passed query for 191699
failed with the exact error:

MCP server 'azure-devops': MCP request failed: Transport closed

Targeted authenticated per-run pagination remained available and was used
successfully. This is not a claim that all Azure DevOps access was unavailable.
The remaining 515 builds have no per-test discovery in this run; the five
queried builds also retain other unprocessed runs/outcomes. Successful and
canceled build inventories are not counted as passing-test coverage.

Three runs were fully paged and reconciled, 9,024 unique result identities:

Build/run Page offsets and sizes Passed Failed NotExecuted Total
191699 / 3710387 0/500, 500/1000, 1500/1000, 2500/1000, 3500/1000, 4500/1000, 5500/500 5,760 1 190 5,951
191931 / 3713578 0/500, 500/500, 1000/500, 1500/500 1,407 1 125 1,533
191916 / 3713346 0/1000, 1000/1000 1,412 0 128 1,540

Run 3710387's server statistics contain two separate Passed entries:
5,758 ordinary and 2 with resultMetadata: flaky; their sum is 5,760.
Both entries were preserved, not overwritten by outcome-key deduplication.
Its zero-incomplete Completed state does not make skipped results passes.
Run 3713578's last page has 33 results: 31 passed and 2 not executed.
Other build-helper pages are not represented as fully enriched/paged inventories.
Identities are deduplicated by (buildId, runId, resultId), never title alone.

Run/result identity, build linkage, outcome and JSON/HTTP 200 were verified
through only the permitted single-run, single-result and build-timeline public
REST forms. Discovery and pagination remained authenticated MCP.
No REST listing or writes, account changes, credential extraction, pipeline
changes or manually queued validation builds were used.

Candidates not added:

  • Microsoft.Data.SqlClient.UnitTests.SimulatedServerTests.SNICloseRaceDeadlockTest.ResponseCompletionRacesClose_DoesNotDeadlock(disposeInsteadOfClose: False):
    failed in 191888 / 3712734 / 100642, reporting Close not completing
    within 30 seconds. This is an explicit deadlock regression test; its test
    and MARS implementation changed between the Fix async cancellation failing to send TDS attention signal #4435 revisions. Windows passes
    are not a comparable Linux control. Product regression versus intermittency
    remains unresolved; no quarantine or repair is proposed.
  • Microsoft.Data.SqlClient.Tests.SqlAuthenticationProviderManagerTests.DefaultAuthenticationProviders_AppConfig:
    191699 / 3710477 / 100098, Assert.IsType expected the dummy provider
    but got null. No exact comparable pass was verified; unconfirmed.
  • Canceled release/6.1 build 191848 exposed the historical
    Microsoft.Data.SqlClient.ManualTesting.Tests.XEventsTracingTest.XEventActivityIDConsistentWithTracing cases
    (SELECT @@VERSION / Text / sql_statement_starting, and sp_help /
    StoredProcedure / rpc_starting). Main now has separate, changed methods;
    those old cases cannot justify marking current methods. Unconfirmed.
    Discovered failed IDs: Text case
    3712220 / 101763;
    RPC cases
    3712228 / 101798
    and
    3712204 / 101798.
  • The same canceled build's
    Microsoft.Data.SqlClient.UnitTests.SimulatedServerTests.ConnectionFailoverTests.TransientFault_WithUserProvidedPartner_ShouldConnectToPrimary(errorCode: 40613)
    failure
    3712193 / 102405
    has no verified same-configuration passing control. Unconfirmed.
  • Already-quarantined observations include ApiShould.TestSqlCommandCancel
    (ExecuteReader and ExecuteNonQuery; truncated connection arguments),
    SqlSchemaInfoTest.TestGetSchema (both Boolean cases), and
    Microsoft.Data.SqlClient.UnitTests.SimulatedServerTests.ConnectionFailoverTests.NetworkError_WithUserProvidedPartner_RetryDisabled_ShouldConnectToFailoverPartner
    / Microsoft.Data.SqlClient.UnitTests.SimulatedServerTests.ConnectionFailoverTests.NetworkError_WithUserProvidedPartner_RetryEnabled_ShouldConnectToFailoverPartner.
    No extra marking, removal or probation change is made. Truncated old case
    titles are not treated as exact current-case matches.

The unconfirmed candidates' additional result/job/timestamp enrichment and
passing-control discovery remain unfinished, not implicitly verified.
Build 191888 used revision 548b08c874059b4f58d52bca65623d7e0d3e929e;
canceled release/6.1 build 191848 used
ac75c52ac351f798e437328f47cf805d8a376b1e.

Checkpoint NOT ADVANCED. The next tagging run must rescan at least its full
14-day window unless a genuinely complete later tagging record establishes a
checkpoint. This run does not supply an exhaustive candidate list or complete
per-test coverage of any build.

Local category validation and execution

Project:
src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj.
Windows, Release, Project references. Commands below were run from this isolated
worktree, not a primary checkout.

dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -f net10.0 --no-restore -c Release --list-tests --filter "FullyQualifiedName~SplitPacketTest&category=flaky" -v:q

Initial attempt: failed NETSDK1004, missing this worktree's assets file.
Retried with restore permitted by that failure:

dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -f net10.0 -c Release --list-tests --filter "FullyQualifiedName~SplitPacketTest&category=flaky" -v:q
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -f net462 --no-restore -c Release --list-tests --filter "FullyQualifiedName~SplitPacketTest&category=flaky" -v:q

Both built and passed discovery. Each selects exactly FullHeaderTest
(existing control) and MARSSplitTest (new quarantine).

dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -f net10.0 --no-build --no-restore -c Release --list-tests --filter "FullyQualifiedName~SplitPacketTest&category!=failing&category!=flaky&category!=interactive" -v:q
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -f net462 --no-build --no-restore -c Release --list-tests --filter "FullyQualifiedName~SplitPacketTest&category!=failing&category!=flaky&category!=interactive" -v:q

Both passed discovery: exactly OneByteSplitTest, AlmostFullHeaderTest,
HeaderPlusOneTest and MARSReplicateTest, with neither quarantined fact.
These are real xUnit/VSTest selections, not source-text-only validation.

dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -f net10.0 --no-build --no-restore -c Release --filter "FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSSplitTest&category=flaky" -v:q --logger 'trx;LogFileName=mars-net10.trx' --results-directory 'C:\Users\mdaigle\.copilot\session-state\83c8f312-cbf8-435c-8fdd-75a34f1d32ca\files\validation' --blame-hang-timeout 2m
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -f net462 --no-build --no-restore -c Release --filter "FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSSplitTest&category=flaky" -v:q --logger 'trx;LogFileName=mars-net462.trx' --results-directory 'C:\Users\mdaigle\.copilot\session-state\83c8f312-cbf8-435c-8fdd-75a34f1d32ca\files\validation' --blame-hang-timeout 2m
git diff --check
  • net10.0: 0 passed, 0 failed, 1 skipped. The three configuration predicates
    reported TargetInvocationException. This is not passing execution coverage.
  • net462: 0 passed, 1 failed, 0 skipped, exit 1. The fact executed and failed
    with the pre-login timeout (initialization=4; handshake=4749).
    It is not a local passing or repair claim. No configuration/body repair was attempted.
  • git diff --check: passed. The only source diff is the new trait and observed-error comment.

Related proposals, publication and release target

Read-only deduplication/context: #5038
is the distinct pending certificate quarantine;
#5039 is separate fixes/probation work.
#4980 and
#4996 are live-verified merged
historical inputs. Prior merged category proposals #5006, #5016, #5020, #5021,
#5024, #5028 and #5047 are already represented upstream. #5047 was read only,
not modified. No open equivalent MARSSplitTest proposal was found in the
pre-publication overlap recheck.

Upstream main at the release-target recheck:
4b27ecc60781e78a26ae0e5f01eb414af676f1f5.
Its canonical SqlClientNextVersion is 8.1.0-preview1; the exact existing
open milestone is #108. The one-time 8.0.0-preview1 / #101 exception
no longer applies. Required publication metadata: milestone 8.1.0-preview1,
existing label Area\Tests, and the exact automated-disclosure first line.
No human milestone override was encountered on this new proposal.

Live publication verification: the PR is open, source and target repositories
are both dotnet/SqlClient, target is main, and the pushed source is under
dev/mdaigle/. The remote diff is exactly one file, two additions, zero deletions,
at full commit eda177d9766b2e860b7cb74030e8dbf20a7a4dfe. The first-line
disclosure, Area\Tests, milestone 8.1.0-preview1 / #108, and the description's
link to this single collapsed run record were all read back and verified.
No merge or auto-merge was enabled.

The initial pre-assignment milestone validation failed at 14:18:07Z; subsequent
automatic validations succeeded. The latest
Validate milestone
completed successfully at 2026-11-13T14:18:45Z, and
Check milestone matches next SqlClient version
succeeded at 14:18:43Z. No check was manually rerun or bypassed.
Other PR validation/review gates are not claimed complete.

All changes are confined to the automation's isolated worktree and its upstream
dev/mdaigle/ branch. No fork is used. Fixes and all probation lifecycle work
are outside this automation's scope.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Review remains blocked because the required trusted, immutable-source policy references were not supplied, and worktree copies cannot substitute.

No open issues 🎉

What changed in this PR

Quarantines the intermittent MARSSplitTest using the existing flaky-test mechanism, without changing test logic, product code, or pipelines.

Changes:

  • Adds a method-level flaky category affecting both net462 and net10.0.
  • Adds a comment documenting the observed pre-login handshake timeout.
  • Preserves the existing FullHeaderTest quarantine and four ordinary sibling tests.
File Description
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​SQL/​SplitPacketTest/​SplitPacketTest.cs Adds the quarantine trait and observed-error comment to MARSSplitTest.

Review effort: Balanced

@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.33%. Comparing base (17bef7b) to head (eda177d).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5065      +/-   ##
==========================================
+ Coverage   68.99%   69.33%   +0.33%     
==========================================
  Files         314      314              
  Lines       45857    69744   +23887     
==========================================
+ Hits        31638    48354   +16716     
- Misses      14219    21390    +7171     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 69.33% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants