UI tests phase 3 - #128
Draft
graeme wants to merge 11 commits into
Draft
Conversation
BrewApp.activate and BrewUITestCase's launched-app tracking both solve problems that belong to running the app under XCUITest on macOS, not to the stubbed wiring the harness happens to launch: a launch that leaves the app windowless has an empty accessibility tree whatever it is wired to, and a leaked process blocks the next test's launch outright. A live suite launching production wiring needs both and can share neither while one is private and the other only reachable through a scenario. activate loses `private`, and tracking gets a name a subclass can call.
The deterministic suite proves our code behaves given an answer; nothing yet proves the answers themselves still arrive. This harness launches the app with no -uiTesting argument at all, so BrewApp.init() falls through to .live() on both seams — the login shell resolves, the machine's real brew runs, and URLSession.shared fetches the real catalogue. The only injection is Homebrew's determinism environment, which is the difference between a reliable canary and a flaky one: without HOMEBREW_NO_AUTO_UPDATE an install can spend minutes updating the tap first, and a test that hangs reads as a broken app. BrewCommandService runs the subprocess with Environment.inherit and a login shell adds to exported variables rather than clearing them, so they survive `-l -i`. State is arranged and cleaned up by shelling out to real brew rather than by driving the UI, so a broken arrange reads as a broken fixture instead of as a red assertion three screens into the flow under test. That helper is deliberately not the code under test: the app reaches brew its own way, and the tests act through that path only. It resolves the executable the way BrewExecutableLocator does, spelled again rather than imported because a fixture actuator resolving brew differently from the app is a difference worth being able to see. Timeouts are an order of magnitude above the stubbed suite's because the work behind them is real. They are generous on purpose: off the PR path, a slow pass costs minutes, while a tight bound costs a canary that cries wolf about the network.
Six canaries over install, uninstall, config, search, navigation and console, reusing the page objects the stubbed suite uses so the only difference between the tiers is what is behind them. If Homebrew changes its JSON shape or CLI output and breaks a happy path, these go red while the fixture-fed suite stays green — which is the entire point, since a fixture cannot notice that it has started lying. Assertions read shape, never values. A row exists, a HOMEBREW_VERSION entry is present, presence and absence transition, output streamed and ended in success. A canary that fails on a version bump gets muted within a week, and a muted canary reports nothing. Upgrade and doctor are absent by design: upgrade needs a deterministically outdated installed package, which a live machine cannot guarantee without brittle bottle pinning, and doctor's output is entirely machine-state dependent so it has no stable happy path. Both stay at Tier 2, where they are cheap and deterministic. The mutating pair owns `hello` — no dependencies, pours in seconds — and force-uninstalls it before and after every test, so the two are order-independent and a failed run leaves no residue. Cleanup runs after the app has been terminated, so it never contends with a running app for Homebrew's lock. Both live tests filter the list through the search field before asserting on a row. A real machine's inventory is long and List virtualizes, so an installed package's row may never render and would fail an assertion that is otherwise correct. The stubbed suite never had to care; its fixtures are short enough to render whole.
They mutate real machine state and depend on the network, so a red run does not necessarily say anything about the change under review. Gating every PR on that trains people to ignore the signal. Brew-UI skips the two classes by name; Brew-E2E selects only them and gives them what live work needs — serial execution, per-test allowances measured in minutes, and retry-on-failure of one. One retry absorbs a transient blip; more would start hiding a real contract break, which is the only thing this suite exists to find. Both plans hang off the existing Brew-UI scheme because xcodebuild -testPlan can only reach a plan the scheme lists.
A canary nobody runs is a canary that reports nothing, so the suite gets a schedule rather than only a plan: nightly on an ephemeral GitHub runner, which ships Homebrew and has egress to formulae.brew.sh and ghcr.io, and by hand before cutting a release — the moment you most want to know brew has not shifted under the app. The release workflow is deliberately not wired to it; that stays a decision someone makes. Concurrency is keyed on the workflow rather than the ref, because two runs would fight over one machine's brew state. scripts/test-e2e mirrors scripts/test-ui, including its refusal to pass -derivedDataPath, and leads with what it does to the machine it runs on: this one installs and uninstalls a formula, and the script is the last place that fact can be read before it happens.
The suite is one script away from anything running in this repo, and its name is one letter from the deterministic one. An agent or a contributor reaching for "the UI tests" must not discover the difference by watching a formula get uninstalled, so AGENTS.md says which is which and that this one is never a quality gate. The memory entry keeps the decisions that the code cannot show: why upgrade and doctor are excluded, why `hello` is the canary and nothing may depend on it, why assertions read shape rather than values, and why the live tests filter a list the stubbed ones never had to.
The suite could not run at all: macOS put up "BrewUITests-Runner is damaged and can't be opened", and xcodebuild sat there until it gave up with "The test runner hung before establishing connection." The dialog is the cause and the hang is only the consequence — a runner Gatekeeper refuses to start never connects. CODE_SIGNING_ALLOWED=NO, copied here from scripts/test-ui, does not produce an unsigned runner. BrewUITests-Runner.app is a copy of Xcode's XCTRunner.app template, which arrives signed by Apple; the build inserts our .xctest into its PlugIns directory and then, with signing disallowed, skips the CodeSign step. What launches is a bundle carrying Apple's signature over contents that no longer match it, which codesign reports as "code has no resources but signature indicates they must be present". Allowing signing with identity "-" ad-hoc signs the runner as sh.brew.BrewUITests.xctrunner with a valid seal. Ad-hoc needs no identity, team or provisioning profile, so this is not a local-only accommodation — it works on a bare CI runner too. scripts/test-ui and the ui-test CI job still carry the flag and the same latent defect; TROUBLESHOOTING.md now says so, so the next person to meet this dialog reads it rather than re-derives it.
The live suite's comments restated what the code says and argued its own design at length. What survives is the part a reader cannot recover: that a login shell keeps exported variables, that a real inventory virtualizes so an installed row may never render, that cleanup runs after the app is terminated to stay clear of Homebrew's lock, and why the runner must be signed. The reasoning behind the rest is in the commits and in E2E/README.md, which is where it belongs.
graeme
force-pushed
the
ui-tests-phase-3
branch
from
August 23, 2026 08:14
0ec251e to
f0f9964
Compare
Two gaps in a suite that retries. A failure's message says an element was missing; it cannot say whether a dialog was covering the app, which is how the last one actually failed. And a retry that turns red into green leaves a run that reads as clean. BrewUITestCase now attaches a screenshot of the whole screen to every issue it records, so the picture lands in the result bundle CI uploads. It goes in as PNG data rather than as the XCUIScreenshot, which cannot cross out of the main actor into this nonisolated override. Both the attachment and the plan keep screenshots always: the interesting one belongs to a failed attempt of a test that ultimately passed, and deleteOnSuccess prunes exactly that. The plan now retries a failing test twice rather than once, and annotate-flaky-tests reads the result bundle afterwards and names any test that needed more than one run to pass — a ::warning:: annotation on GitHub, plain text locally, never a failure by itself. It also tells the workflow to keep the bundle for a green-but-retried run, so the evidence the annotation points at still exists.
Nightly cron only fires on the default branch, so the suite would have gone unrun until after it merged, and the first real evidence it works would have arrived on main. Moving it to pull_request means it is exercised where it can still be fixed. It stays a separate job rather than joining the deterministic suite's: Brew-UI still skips these tests by name, so a live failure reads as a live failure and never as a broken fixture-backed test. Concurrency now keys on the pull request, since queueing every PR behind one run only made sense while the group modelled a single shared machine. The workflow file loses its `_nightly` name, and the docs that promised a nightly schedule now say what actually happens.
The suite documented itself at essay length: paragraphs restating what the next line does, and reasoning that belongs in a commit message. Reading a page object should not mean reading an argument for page objects. What is left is one line wherever one line does: the surprises (a `.searchable` field that cannot carry an identifier, `List` row text arriving as accessibility value rather than label, an Uninstall button below the fold, a fixture bucket that must be absent rather than empty) and the type-level sentence saying what each file is for. Everything that merely restated a declaration is gone. 96 comment lines removed across the suite; no code changed.
graeme
force-pushed
the
ui-tests-phase-3
branch
from
August 23, 2026 09:21
f0f9964 to
be6c3dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Live end-to-end canaries (Tier 3)
Summary
Adds a small suite that runs the app with production wiring against real Homebrew and the real network, plus the test plan, script and schedule to run it. Tier 2 answers from fixtures, so it cannot notice when those fixtures have started lying; this suite can.
Changes
Live suite (
BrewUITests/E2E/)BrewE2EApplaunches with no-uiTestingargument, soBrewApp.init()falls through to.live()on both seams: real login shell, real brew, realURLSession.shared, real bottles. The only injection is Homebrew's determinism environment.Brewis aProcesshelper used to arrange and clean up state, never as the code under test. Tests act through PR 2's page objects unchanged.hello, force-uninstall it before and after every test, and are order independent. Nothing else may depend on it.Wiring
Brew-E2E.xctestplanandscripts/test-e2e: serial, minute-scale allowances, one retry. Skipped by name inBrew-UI.xctestplan.e2e_nightly.ymlruns it nightly on an ephemeral runner. The release workflow is untouched.BrewApp.activateand app tracking inBrewUITestCaseare now reachable by a second launcher.E2E/README.md, a TROUBLESHOOTING entry, an AGENTS.md warning, a memory entry.Why this split
Upgrade and doctor are excluded by design: upgrade needs a deterministically outdated package and doctor's output is entirely machine dependent, so neither has a stable happy path live. Both stay at Tier 2. Keeping the suite off the PR path means the pipeline is never gated on live flake.
Testing
scripts/test, swiftformat, swiftlint strict, BrewUILint: all clean.xcodebuild build-for-testingfor the Brew-UI scheme.scripts/test-e2egreen locally.Notes for reviewers
The E2E script allows code signing where
scripts/test-uidisables it. With signing disallowed the runner keeps Apple's XCTRunner signature over contents the build has replaced, and macOS refuses to launch it as damaged.scripts/test-uiand theui-testjob carry the same latent defect, documented but not changed here.PR checklist