[harder]

Supersedes #5736.

Markdig 1.4.2 recognizes Markdown structures that exposed gaps in Terminal.Gui's renderer: definition terms and empty list items could disappear, and hard breaks became spaces. The dependency audit also found dropped abbreviation labels and table content corrupted by serializing Markdig's AST and reparsing it with a simpler inline parser.

This PR upgrades Markdig from 1.4.0 to 1.4.2, preserves the parsed content through rendering, and exercises the resulting behavior in a published native AOT executable.

Changes

• Preserve definition terms, indent definitions, include definitions in block end-line measurement, and retain empty ordered/unordered list markers.
• Render hard breaks in paragraphs, headings, links, quotes, and lists, including continuation indentation and usable heading anchors. Soft breaks remain spaces.
• Preserve abbreviation labels in text and table cells.
• Carry parsed table-cell segments through measurement, drawing, wrapping, and link hit testing. Embedded backticks, escaped literals, nested link labels, and original URLs survive without being reparsed. Preserve combined bold/italic/strikethrough/link styles and use safe code delimiters in the public text representation.
• Render alert kinds visibly as bold quote headers; cover opt-in nested alerts.
• Add 171 regression cases overall, including rendered cell text and attributes, keyboard/mouse activation, wrapping, data reassignment, and prior Markdig changes around strict GFM normalization, emoji alignment, CJK text, nesting limits, and link/emphasis parsing.
• Extend the existing native AOT smoke app to assert headings, anchors, definitions, lists, hard breaks, abbreviations, code, table content, styled link activation, alert kinds, nested alerts, and optional emoji/CJK pipeline construction.
• Update the Markdown demo and include a PTY recording with actual table-link activation.

Rendering and parsing safeguards

• Consume list bullets once within definition lists and opt-in nested alerts. Keep generic extension-block prefix behavior unchanged.
• Keep hard-break lines valid even when empty; a trailing break ends its line without adding a phantom continuation line. Exercise selection and hit testing on leading/consecutive breaks.
• Use Markdig's inline parsers and shared cell lowering for standalone table sources. Preserve escaped delimiters, bracketed labels, nested emphasis, code pipes (including code inside link labels), original URL destinations, and unmatched backtick runs across rendering and clipboard reparsing. Preserve cell paragraph/soft-break separation and encode explicit cell hard breaks as newline entities. Grid-cell pipes remain escaped in clipboard pipe tables. Preserve unsupported leaf inlines, such as math, as literal source rather than dropping their text; standalone parsing retains source spans for this fallback.
• Document TableData cells as Markdown inline source. Preserve literal syntax with original inline source where possible, leaving harmless snake_case and [1] readable in the default pipeline. Reuse original source spans for custom-pipeline plain cells when the standalone parser preserves their text, styles, and link behavior; ordinary punctuation such as Hello, world. stays readable on copy. Keep entity encoding where needed to prevent other extensions from changing literal text, including literal domains becoming autolinks. Respect escaped row boundaries and retain empty cells. Serialize resolved links and images with angle-bracket destinations and their original titles, including spaces, quotes, and pipes. Preserve literal ! before links so reparsing does not turn links into images. Unescaped generic attributes remain metadata, matching Markdig; escaped attributes remain visible text.
• Preserve every segment field through slicing and highlighter conversions. Keep one consistent set of InlineRun record properties and verify equality and additional-style round trips.
• If Markdig parsing raises ArgumentException, log a warning with the original exception and lower literal paragraphs through the normal state-reset path, preserving blank source lines. Content invalidation clears old anchors; layout rebuilding repopulates their positions. Recovery resets slugs and caches the parsed result. Markdig 1.4.2 reports nesting failures as generic ArgumentException; retaining parse-only recovery without matching its message is intentional. Exceptions from AST lowering remain outside the catch.
• Reconstruct alert markers using their original kind casing when complete headers are selected. Keep displayed kinds uppercase and partial selections faithful to visible text.
• Initialize table data, retained segments, and the intended width together. Use original code source or a character loop for fallback backtick delimiters; remove the newly introduced compiled regex fields. Cache column min/max width bounds when retained content is built, so width-only relayout does not remeasure every cell. Match tab/newline word boundaries between measurement and wrapping. Wrap complete words across emphasis and link segments while preserving their metadata and keeping separately styled combining marks with their base characters; keep wide graphemes intact in narrow columns without inventing blank lines before hard breaks. Scan separators without a per-word character array or a separate newline-only segment branch. Remove unused internal string measurement overloads; test the segment measurements used by production. No public measurement APIs are removed.
• Use layout/plain-driver buffer tests for text, styles, and mouse activation; retain one application case specifically for keyboard focus. Replace private reflection in PR-added tests with typed internal algorithm calls, retained content, or visible buffer/selection assertions.

The existing public TableData interface and standalone string-based MarkdownTable input remain available. This PR preserves parsed table content when rendered through Markdown; it does not introduce the larger table geometry API.

Dependency audit and follow-up

Release-history audit for the Markdig 1.4.2 upgrade

Dependency history

The merged Markdown view first referenced Markdig 0.39.0 in commit c6d67e362
(#4955, 2026-04-17). The subsequent package bump commits below changed only
Directory.Packages.props; they did not add release-specific regression tests.
Existing CI coverage alone therefore does not establish coverage of each upstream fix.

Date	Terminal.Gui commit	Version change
2026-04-18	0cde267fd (#4989)	0.39.0 → 1.1.2
2026-04-20	0c7dee4b4 (#5017)	1.1.2 → 1.1.3
2026-06-15	8130f52f0	1.1.3 → 1.3.0
2026-06-17	676c9ec25	1.3.0 → 1.3.1
2026-06-19	491fadafb (#5503)	1.3.1 → 1.3.2
2026-09-21	22ef38a6d (#5685)	1.3.2 → 1.4.0
PR #5736	56efec662	1.4.0 → 1.4.2

The May 5 package maintenance commit also mentions Markdig but retains 1.1.3.
The open 1.4.1 dependency PR is not a merged version transition.

Scope and results

Reviewed upstream release notes from 0.39.1 through 1.4.2, relevant parser and
extension source, and upstream regression examples against Terminal.Gui's AST
lowering, table conversion, layout, ANSI rendering, and native smoke app.
The new regression tests exercise Terminal.Gui output and table data; they do
not attempt to duplicate Markdig's entire CommonMark test suite.

Upstream versions	Relevant changes	Terminal.Gui result
0.39.1–0.43.0	Grid bounds, definition-list parser safety, CommonMark 0.31.2, pipe tables after paragraphs, multiline inline code and backticks	Parser fixes apply automatically. Added integration assertions for open grid cells, tables after paragraphs, and code in table cells. Existing inline-code tests remain applicable.
0.44.0–0.45.0	Link normalization, supplementary Unicode, linear-time pipe parsing	Unicode is retained through the renderer. Supplementary-character coverage added. Parser performance applies automatically; Terminal.Gui still performs its own layout work.
1.0.0–1.1.2	Removed obsolete APIs, AST authoring APIs, CJK emphasis, emoji in table cells, reference source spans	Build verifies the APIs we use. Added custom-pipeline CJK and emoji table tests. AST mutation, source-span, HTML diagram and KaTeX changes do not add terminal rendering capabilities automatically.
1.1.3	Abbreviations corrupting adjacent emphasis	Found an additional integration bug: Terminal.Gui discarded AbbreviationInline labels. Fixed both normal text and table extraction, with tests for labels before and inside emphasis.
1.2.0–1.3.0	Optional nested alerts, CJK entities, unmatched subscript in tables, configurable nesting limits, ordered lists in quotes	Added custom-pipeline nesting-limit and unmatched-subscript table assertions. Existing quote/list tests remain applicable. Nested alerts remain opt-in.
1.3.1–1.3.2	Non-ASCII slug normalization, superscripts, emoji table alignment	Added an emoji alignment/content test. Terminal.Gui generates its own heading slugs and does not call LinkHelper.Urilize. Superscripts continue using terminal emphasis rather than raised glyph positioning.
1.4.0	Optional strict GFM tables, nested link labels, unmatched emphasis closers, emoji/emphasis interaction	Added strict-GFM column normalization, nested link label, and trailing-text regression tests. The default permissive table syntax remains unchanged.
1.4.1–1.4.2	Definition syntax and continuation, empty lists, HTML boundaries, setext headings, AOT metadata	Fixed dropped definition terms and empty item markers. Hard breaks now produce rendered lines with continuation indentation. Added parser-to-renderer regression tests and executable AOT assertions.

Deliberate boundaries and follow-up opportunities

• To enable strict GFM tables, supply a MarkdownPipeline built with
  UsePipeTables (new PipeTableOptions { UseGfmRules = true }). Integration tests
  confirm row padding and truncation. Switching the default would change accepted syntax.
• To enable CJK-friendly emphasis, emoji shortcodes, or nested alerts, supply a
  custom pipeline with the corresponding extension options. These are optional
  upstream features; this PR does not change the default pipeline.
• InferColumnWidthsFromSeparator (0.41.0) produces Markdig column-width metadata,
  but TableData currently carries only text and alignment. Honoring those widths
  needs a separate table-sizing API and layout design.
• Issue #5738 tracks inferred
  sizing hints and richer table geometry, with design questions and acceptance criteria.
• Grid-table row/column spans and arbitrary block content exceed the current
  MarkdownTable model. The first grid row is displayed as its header. This PR
  verifies content preservation for the upstream open-cell regression, not full grid fidelity.
• Tables rendered by Markdown now carry parsed inline segments through layout,
  drawing, and link hit testing, preserving code, escaped literals, combined styles,
  and original link targets. Their public text representation remains available
  for compatibility and clipboard reconstruction. Standalone string-based
  MarkdownTable input uses Markdig inline parsing and the same lowering as AST-backed cells.
• Alert kinds are now visible as bold quote headers. Nested alert parsing remains
  opt-in; dedicated per-kind colors are not introduced.
• FindClosestBlock / FindBlockAtPosition fixes concern reference/footnote groups.
  Terminal.Gui does not call them and discards the parsed document after lowering.
  Source-to-preview navigation would first require retained source mappings.
• Upstream HTML, normalization, and round-trip renderer changes are not directly
  consumed: Terminal.Gui parses Markdown and renders the AST itself.

Terminal.Gui began with Markdig 0.39.0 and subsequently moved through 1.1.2, 1.1.3, 1.3.0, 1.3.1, 1.3.2, and 1.4.0. The dependency bump commits added no release-specific regression tests. This audit reviews the intervening releases and adds coverage at the parser-to-renderer boundary.

Follow-up #5738 tracks inferred column sizing and richer table layout, including headerless/multiple-header-row grid tables, row/column spans, block content, resize/overflow policy, navigation, selection, and AOT coverage. It includes examples, implementation pointers, design decisions, and acceptance criteria.

Strict GFM tables, CJK-friendly emphasis, emoji shortcodes, and nested alerts remain opt-in pipeline choices. Source-to-preview navigation also remains separate work. Preserving Markdown hard-break syntax in partial paragraph selections remains deferred; hard breaks render correctly, and full-document copying retains the original source.

Validation

• Restore and full Debug solution build passed. Warning diagnostics match the prior build; no new warnings were introduced.
• Completed the POST-GENERATION-VALIDATION.md checklist for all changed code, including added-line spacing/brace/whitespace scans and manual review. The PR diff has no added .ToArray () or .ToList () calls.
• Full parallel suite passed: 18,195 passed, 17 skipped, including all 552 Markdown cases. The terminal environment used NO_COLOR unset, TERM=xterm-256color, and COLORTERM=truecolor.
• All 35 nonparallel tests passed.
• Published native Linux ARM64 using GCC; the expanded Markdown assertions and full application lifecycle passed.
• Recorded with tuirec; verified text and the original activated URL in the asciinema cast and inspected the rendered frame.

The recording is retained here as PR evidence, using an immutable historical asset URL; it is no longer part of the source tree.

To pull down this PR locally

git remote add copilot https://github.com/harder/Terminal.Gui.git
git fetch copilot codex/markdig-rendering-fixes
git switch --detach copilot/codex/markdig-rendering-fixes