Skip to content

Fixes #5680. Begin() without Run() leaves MainLoopSyncContext installed, and Post() then queues continuations nothing drains - #5722

Open
BDisp wants to merge 19 commits into
tui-cs:developfrom
BDisp:issue/5680_app-begin-mainloopsynccontext
Open

BDisp wants to merge 19 commits into
tui-cs:developfrom
BDisp:issue/5680_app-begin-mainloopsynccontext

Conversation

@BDisp

@BDisp BDisp commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes

Summary

  • Fix callback pumping for Begin-only sessions so posted work is processed while a session is active.
  • Preserve safe thread-pool fallback behavior and run-loop state handling.
  • Restore synchronization context behavior consistency for both Post and Send.
  • Keep exception and blocking semantics intact.

Changes

  • Updated dispatch flow in ApplicationImpl.Dispatch to process posted callbacks correctly during Begin/End lifecycle.
  • Adjusted run/session handling in ApplicationImpl.Run to preserve expected state transitions.
  • Refined MainLoopSyncContext behavior for context restoration and dispatch semantics.
  • Added regression coverage in ApplicationImplBeginEndTests for await continuations, UI event thread affinity, invoked actions, and cleanup after End.

Testing

  • Passed: ApplicationTests.BeginEnd.ApplicationImplBeginEndTests.Await_After_Begin_Always_Resumes

Pull Request checklist:

  • I've named my PR in the form of "Fixes #issue. Terse description."
  • My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
  • My code follows the Terminal.Gui library design guidelines
  • I ran dotnet test before commit
  • I have made corresponding changes to the API documentation (using /// style comments)
  • My changes generate no new warnings
  • I have checked my code and corrected any poor grammar or misspellings
  • I conducted basic QA to assure all features are working

Pump posted callbacks for Begin-only sessions while preserving safe thread-pool fallback and run-loop state. Restore synchronization context consistently for `Post` and `Send` while maintaining exception and blocking semantics. Add coverage for await continuations, UI events, thread affinity, and session cleanup.
Copilot AI balanced review requested due to automatic review settings October 15, 2026 18:41
@BDisp
BDisp requested a review from tig as a code owner October 15, 2026 18:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@BDisp BDisp changed the title Fix await resumption after Begin Fixes #5680. Begin() without Run() leaves MainLoopSyncContext installed, and Post() then queues continuations nothing drains Oct 15, 2026
BDisp added 12 commits October 15, 2026 21:42
Prevent posted work from starting nested rendering while inline output is settling or `LayoutAndDraw` is active. Add coverage ensuring captured await continuations remain pending until rendering completes and dispatches are explicitly drained.
Pass the current test cancellation token to four `Task.Run` continuations in `BeginEndTests.cs` while preserving existing toolkit continuation behavior.
Add tests verifying IsRunLoopRunning is false after initialization, remains true across nested run loops, and becomes false only after the final runnable exits.
Update `IApplication.Begin` and multitasking documentation to explain that Begin prepares a session without starting the continuous event loop. Document limited async continuation support during the first active Begin-only session and clarify that `Run` remains required for ongoing input and UI processing.
Add tests verifying callback exceptions propagate through `Send` and pending synchronous sends complete without deadlocking when the run loop stops. Include ANSI application setup, thread coordination, timeouts, assertions, and resource cleanup.
Assert timer processing starts on the worker thread and the callback is subsequently marshaled to the application’s main thread.
Cover `IsRunLoopRunning` reset after normal stop requests and exception-driven exits, including active-state verification during iteration.
Clarify queued dispatch ownership during session end:
active sessions retain only owner work, while final teardown
releases remaining callbacks to the thread pool.

Add a 25-iteration concurrency test verifying
SynchronizationContext.Send executes at most once, completes
after shutdown, and leaves no queued dispatches.
@harder

harder commented Oct 19, 2026

Copy link
Copy Markdown
Member

Hi @BDisp, thanks for working on this. I want to make sure I understand the design decision correctly.

In the issue, we discussed that Post could fire on a worker thread, so running RunIteration there could end up processing input and drawing the UI off-thread — and had a few different ideas of how to handle that.

This PR implements the RunIteration-from-Post approach. I had my LLM trace through PumpPostedWork and confirmed it does run the full iteration (including the Iteration event, layout/draw, and timers) on the posting thread, blocking it for ~40ms per frame. That's the thread-affinity problem we were trying to avoid.

A few other questions:

  1. Begin + Run leftover context: This PR doesn't address the earlier finding where Begin/Run leaves MainLoopSyncContext installed. Should that be a separate fix?
  2. Second Begin session: The !HasEndedSession condition on the pump means a second Begin after any End still strands posts (sticky flag). Is that intended?

I may be misreading the code or the intent. But I think we should revisit whether pumping on the posting thread is the right path.

@BDisp

BDisp commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @harder. It was indeed a different approach than before, but with the same consequence. The iteration never actually happens in UIThread and it's not in accordance with the thread-safe protocol.
Disregard this PR as I recognize it was a futile attempt, but it was instructive.
Therefore, it's proven that without Run, it's not possible for posted threads to execute in UIThread.

@BDisp
BDisp marked this pull request as draft October 19, 2026 23:51
BDisp added 3 commits October 21, 2026 00:36
context, so a Begin without Run left a context nothing drained. Posts
from await continuations were queued and never ran.

Move installation and restoration into Run:

- Run saves the caller's context, installs the app's before Begin, and
  restores it in a finally, only if the app's context is still current.
  A nested Run saves the app's own context, so only the outermost Run
  restores the caller's.
- Begin and End no longer read or write SynchronizationContext.Current.
- Remove _callerSynchronizationContext, which was shared mutable state
  written on the Begin thread and read on whichever thread called End,
  and drop its use in ResetState.
- Init creates the MainLoopSyncContext inside the _dispatchLock block,
  before Initialized = true, so a reader that sees Initialized also sees
  the context.

Work posted from other threads still runs only on the UI thread, when it
iterates. Leftover callbacks are still released to the thread pool when
the final session ends (tui-cs#5636).

Docs: update IApplication.Begin remarks, application.md and
multitasking.md to say Begin starts no loop and installs no context, and
to use Invoke to reach the UI thread after an await.

Tests: add SyncContextLifecycleTests covering Begin without Run,
Run with one runnable and nested runnables, an existing caller context,
and restoration after an exception. Update tests that relied on Begin
installing the context.
Pressing Esc while "Cancelable Load Items" was running ended the session
without canceling the load. The await continuation then resumed on a
thread-pool thread after shutdown and set the button text on a disposed
view, throwing ObjectDisposedException (DEBUG_IDISPOSABLE).

- Capture the app, button and list in locals at the start of
  CallLoadItemsAsync instead of reading fields after the await, which a
  later run could reassign.
- Marshal the post-await UI updates through InvokeAsync bound to the
  session owner, as MethodAsync already does, and catch
  OperationCanceledException and NotInitializedException for the case
  where the session ended or the app was disposed first.
- Create the CancellationTokenSource only when a load starts. The Cancel
  click now cancels the running load instead of replacing the source.
- Pass the token to LoadItemsAsync and Task.Delay so a canceled load
  stops promptly.
- Cancel the in-flight load when Run returns, since Esc and Quit end the
  session without pressing the button.
@BDisp

BDisp commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @harder, thanks for tracing it, you read the old code correctly. Pumping on the posting thread was the wrong approach. Posts can come from any thread, so it ran input, layout and drawing off the UI thread, and it also had no gate against concurrent pumps. I've dropped it and followed the direction you suggested in the issue, so the PR now takes a different approach.

What changed

  • Run owns MainLoopSyncContext. It saves the caller's context, installs the app's before Begin, and restores it in a finally, only if the app's context is still current. A nested Run saves the app's own context, so only the outermost Run restores the caller's.
  • Begin and End no longer read or write SynchronizationContext.Current. A Begin-only session leaves the caller's context alone, so an await after Begin completes normally instead of posting to a queue nothing drains.
  • The pump (PumpPostedWork, _isPumpingPostedWork, the CanStartUiWork change) is gone, and ApplicationImpl.Dispatch.cs and MainLoopSyncContext.cs are back to develop. Work posted from other threads runs only on the UI thread, when it iterates, and leftovers are still released to the thread pool when the final session ends (IApplication.RunAsync deadlocks after await on ambient SynchronizationContext #5636).
  • I removed _callerSynchronizationContext from Begin, End and ResetState. It was shared mutable state written on the Begin thread and read on whichever thread called End. Init now creates the context under _dispatchLock before setting Initialized.

Your questions

  1. Begin + Run leftover context. This PR now addresses it, since restoration lives in one place (Run's finally) instead of being split across Begin, End and ResetState. I added tests for single and nested Run, a pre-existing caller context, a context replaced during the run, and restoration after an exception.
  2. Second Begin after an End. This no longer applies. The sticky !HasEndedSession condition existed only for the pump and is gone, and Begin no longer installs a context, so a later Begin can't strand anything.

Behavior and docs

Without Run, nothing drains work posted from other threads, so the Begin XML doc, application.md and multitasking.md now say that Begin starts no loop and installs no context, and that Invoke is how to reach the UI thread after an await. I removed the tests that asserted the pump behavior.

I also found a related problem in the Threading scenario: Esc ended the session while a load was in flight, and the continuation touched disposed views. I fixed it separately in Threading.cs in its own commit, using _cancellationTokenSource?.Cancel (); to cancel if it isn't null.

@BDisp
BDisp marked this pull request as ready for review October 21, 2026 00:12
@BDisp
BDisp requested a review from harder October 21, 2026 00:12
@BDisp

BDisp commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

PR Review: #5722 – Fixes #5680. Begin() without Run() leaves MainLoopSyncContext installed, and Post() then queues continuations nothing drains

Status: Draft (as of the latest visible state). Author (BDisp) previously marked an earlier approach as futile after review feedback and switched to a cleaner design.

Problem

Begin() was installing MainLoopSyncContext as the ambient SynchronizationContext. Without a subsequent Run() (i.e., no pumping loop), any await / Post / Send continuations queued into that context were never drained → hangs or stranded work. Related issues around context restoration after nested sessions / exceptions and leftover context after Begin+Run were also in play (#5636, etc.).

Solution (final direction in the PR)

Move ambient context ownership entirely into Run:

  • Init still creates the MainLoopSyncContext instance (now inside the relevant lock, before Initialized = true).
  • Begin / End no longer touch SynchronizationContext.Current.
  • Run (and the generic / async variants) captures the caller’s context, installs the toolkit context before Begin, and restores the caller’s context in a finally only if the toolkit context is still current.
  • Nested Runs correctly keep the toolkit context until the outermost returns.
  • Removed the shared mutable _callerSynchronizationContext field (and its use in ResetState), eliminating a potential cross-thread race.
  • Docs updated (IApplication.Begin remarks + multitasking/application docs) to state clearly that Begin starts no loop and installs no context; use Invoke / Run to reach the UI thread.
  • Example scenario (Threading.cs) fixed for a post-shutdown continuation that touched disposed views.
  • Substantial new/updated tests covering:
    • Begin-only sessions leave caller context alone and do not hang on await.
    • Posts during Begin-only are queued and only run when explicitly drained on the UI thread.
    • Single / nested Run, existing caller context, exception paths, restoration semantics.
    • Send racing with End, exception propagation, no deadlocks, at-most-once execution, etc.
    • Timer / dispatch affinity adjustments.

Strengths

  • Correct and minimal design that matches the discussion on the issue: the context should only be ambient while a pumping loop exists.
  • Eliminates shared mutable state that was written on the Begin thread and read on the End thread.
  • Good nested-session and exception-path coverage; the “restore only if still current” check is the right way to handle intentional context swaps inside handlers.
  • Docs and example fixes make the contract clearer for both library users and internal tests.
  • Tests are thorough and exercise the exact failure modes that previously hung.

Concerns / Suggestions

  1. Draft status & earlier approach
    The PR history contains an earlier attempt that pumped RunIteration from Post (which can run on worker threads). That was correctly abandoned. Confirm the final commits are only the “install in Run only” design and that any leftover pump-from-Post code is gone.

  2. Test that previously asserted context installation by Begin
    Several tests were updated (or new ones added) because they relied on the old behavior. Double-check that every place that previously expected SynchronizationContext.Current after Begin has been adjusted, especially non-parallelizable suites and any inline-output / await-continuation tests.

  3. Send / race tests
    The 25-iteration race between Send and End is good. Consider also verifying that a Post that arrives after the final session ends still falls back to the thread pool (the existing “leftover callbacks released to thread pool” behavior).

  4. Documentation completeness
    The Begin remarks now correctly say it does not change the ambient context and that nothing drains work without Run. A short note in the multitasking guide about “if you must use Begin/End directly for testing, either drain manually or use StopAfterFirstIteration + Run” would help future test authors.

  5. Minor style / hygiene

    • Spacing in generic constraints (new () vs new()) appears in a couple of places; match the project’s .editorconfig.
    • A few new tests still contain comments attributing them to specific models (“Copilot – GPT-…”, “Claude – …”). Harmless but can be cleaned up if the project prefers neutral comments.
    • The Threading scenario fix is solid; the extra cancellation + local capture + InvokeAsync is the right pattern.
  6. Public surface
    No public API break is introduced (only behavioral tightening of when the context is ambient). This is a good outcome.

Verdict

Approve once the PR is marked ready and the final commits are confirmed to contain only the “context owned by Run” approach.

The design is the right one, the tests look comprehensive, and the docs make the contract explicit. The earlier dead-end of pumping from worker-thread Post has been correctly discarded. Nice work navigating a subtle threading/lifecycle issue.

…d must not sit on the toolkit queue forever; it falls back to the thread pool (tui-cs#5636 / tui-cs#5680).
@harder

harder commented Oct 22, 2026

Copy link
Copy Markdown
Member

Hi @BDisp, thanks for revisiting this. It's looking good. The revised approach addresses the earlier concerns. Having Run own the synchronization context, while Begin and End leave the caller’s context alone, looks correct. The worker-thread pump is gone, and the nested-run and exception-restoration behavior looks sound.

I have a few smaller follow-ups from an AI code review:

  1. The Threading cancellation change seems incomplete. The loader’s Task.Delay(5000) doesn’t receive the cancellation token, so canceling after Run returns doesn’t stop an already-running load. It can still call LogJob after the views are disposed, including from the canceled branch. I reproduced that late logging. Since this is a separate existing issue, could we leave that hunk out and address it in a follow-up?

  2. Please restore the original thread-affinity assertion. In WorkerTimerPump_CannotRunPostedCallbackOffUiThread, assigning callbackThreadId directly in the worker weakens what the test verifies. Keeping that variable written only by the callback, with Assert.Null before the UI drain, makes the intent clearer. I tried the original assertion against this revision, and it passes.

  3. Small nit: Please refresh the PR description and stale comments. The description still describes the abandoned pump approach, and some test comments still say Begin installs the context. The Begin docs should also clarify that Invoke requires UI iteration - it won’t make an idle Begin-only session process work.

These are focused cleanup requests; I don’t think another architectural change is needed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Begin() without Run() leaves MainLoopSyncContext installed, and Post() then queues continuations nothing drains

3 participants