diff --git a/.claude/skills/forge-ci/SKILL.md b/.claude/skills/forge-ci/SKILL.md index 345fc166..e9b6c7de 100644 --- a/.claude/skills/forge-ci/SKILL.md +++ b/.claude/skills/forge-ci/SKILL.md @@ -176,7 +176,7 @@ Then the two log sources: ```bash # Job log (build phase, staging, packaging errors): -gh api repos//actions/jobs//logs > job.log +gh api --allow-escape-sequences repos//actions/jobs//logs > job.log grep -nE "error:|CMake Error|No matching distribution|FAILED" job.log # On-device test output (the actual pytest run) — it is an ARTIFACT, usually @@ -185,6 +185,13 @@ gh run download --repo -D artifacts/ cat artifacts/test-py3.12---*/console.log ``` +Without `--allow-escape-sequences`, `gh` may refuse (*"the response contains terminal escape +sequences"*), write nothing and exit 1 — and a grep over the empty file is a silent false +negative. console.log does not print the interpreter, so after a `mobile_test_pythons=ALL` +run confirm each leg from its job log: the interpreter, then the wheel pip actually +installed (flet's `-vv` output wraps lines, so join them first): +`grep -oE -- '--python-version [0-9.]+' job.log; sed -E 's/^[^ ]+ +//; s/ +$//' job.log | tr -d '\n' | grep -oE 'Processing */[^ ]*dist-test/-[A-Za-z0-9_.+-]*\.whl'`. + A missing console.log artifact for a failed 3.12 job means the job died *before* the device test — almost always at recipe-tester packaging (resolution: see Chains) rather than on the device. diff --git a/.claude/skills/local-recipe-testing/SKILL.md b/.claude/skills/local-recipe-testing/SKILL.md index 091ae095..dba10d76 100644 --- a/.claude/skills/local-recipe-testing/SKILL.md +++ b/.claude/skills/local-recipe-testing/SKILL.md @@ -92,6 +92,24 @@ DATA=$(xcrun simctl get_app_container "$UDID" com.flet.recipe-tester data) for i in $(seq 1 30); do grep EXIT "$DATA/Library/Caches/console.log" 2>/dev/null && break; sleep 5; done ``` +## Shortcut: the consumer example pass on CI's wheels + +Once a CI run is green, its `wheels-py3.X---*` artifacts are the exact +wheels the device tests used. Flatten them into one find-links dir and the example app +builds with **no local forge env at all** — no support tarballs, no NDK, no `forge` run: + +```bash +gh run download --repo -p 'wheels-*' -D /tmp/ci_wheels +mkdir /tmp/ci_wheels/findlinks && cp /tmp/ci_wheels/wheels-*/*.whl /tmp/ci_wheels/findlinks/ +cd recipes//examples/ +PIP_FIND_LINKS=/tmp/ci_wheels/findlinks uv run flet build apk # or ios-simulator +``` + +The example builds for 3.14 (gotcha #2), so the run must include the 3.14 leg. To read the +app's own output without racing a screenshot against the update, the Flet patch log +carries every control value: `xcrun simctl spawn "$UDID" log show --last 2m --style compact +--predicate 'process == ""'` (a release APK logs no patches; use screenshots there). + ### forge slice syntax (quick reference) `android:arm64-v8a` | `android:x86_64` | `android:armeabi-v7a` | `iphonesimulator:arm64` | `iphonesimulator:x86_64` | `iphoneos:arm64` — the first token is the **SDK**, not the OS. `forge iOS:arm64` dies with a raw `KeyError: 'iOS'` (only the bare-platform forms `forge android` / `forge iOS` take the OS name, and those build every arch). @@ -115,6 +133,12 @@ for i in $(seq 1 30); do grep EXIT "$DATA/Library/Caches/console.log" 2>/dev/nul sed -i '' 's/hw.ramSize=.*/hw.ramSize=6144/' "$cfg" 2>/dev/null || echo 'hw.ramSize=6144' >> "$cfg" ``` (`aosp_atd` is rootable but headless — it can't run a Flet GUI app, so don't use it here.) + If the emulator dies with *"Cannot find AVD system path. Please define ANDROID_SDK_ROOT"* + (or, with that variable set, *"Broken AVD system path"*), the AVD survived but its system + image did not (`system-images/` empty) — exporting the variable only changes the message. + Reinstall the image with the SDK's own sdkmanager; a Homebrew `sdkmanager` first on `PATH` + installs into its own root, where the emulator never looks: + `"$SDK/cmdline-tools/latest/bin/sdkmanager" --sdk_root="$SDK" "system-images;android-34;google_apis;arm64-v8a"`. 5. **Give the emulator RAM + disk.** A heavy `.so` (polars ~130 MB) + Python + the Flutter engine OOM-kills the app on a default AVD (`lowmemorykiller: Kill 'com.flet.recipe_tester'`). 6 GB RAM avoids it. The ~100–235 MB APK install needs a big `/data` (6 GB partition); if you hit `INSTALL_FAILED_INSUFFICIENT_STORAGE`, free space (uninstall old apps) or use a fresh AVD. @@ -126,7 +150,7 @@ for i in $(seq 1 30); do grep EXIT "$DATA/Library/Caches/console.log" 2>/dev/nul 9. **Android `console.log` lives in the app's CACHE dir — `/data/data/com.flet.recipe_tester/cache/console.log` — NOT under `files/flet/app/`** (that's the app code; `python_site_packages` is a SIBLING under `files/flet/`). Polling the wrong dir looks like "the app never wrote a result" and cost ~10 min during the sherpa-onnx validation. Root is still required to read it (gotcha #4): `adb root` then `adb shell cat …`, or `adb shell su 0 cat …` on a google_apis image. -10. **`flet build ios-simulator` resolves the `iphoneos` (device) wheel AS WELL as both simulator ones.** It configures pip for `iphoneos.arm64` + `iphonesimulator.arm64` + `iphonesimulator.x86_64` and needs a wheel for EACH — a partial local matrix fails with `No matching distribution found`. Build all three iOS slices first (for the recipe AND every `flet-lib*` host dep). CI never hits this because it dumps all of `dist/*.whl` into its find-links dir. (`flet build apk` needs only the one `--arch` slice — the asymmetry is iOS-only.) Long-standing gotcha; re-hit during the onnxruntime iOS spike. **Worse: serious_python's PER-ARCH native staging (`build/site-packages//`) resolves those slice wheels from the INDEX directly and does NOT honor `PIP_FIND_LINKS`/dist-test locally** — so a hand-patched `-9999` wheel in your find-links dir is used for the initial pip install but the staged PYTHON code (e.g. `cv2/__init__.py`, whichever slice it picks — often `iphoneos.arm64`) still comes from the published wheel. Net: you cannot validate a *hand-patched loader* on a local `ios-simulator` build; use a real `forge` build of all slices, or verify in CI (where the freshly-built slice wheels ARE used — this is why coolprop iOS passed in CI but a hand-patched opencv wouldn't locally). +10. **`flet build ios-simulator` resolves the `iphoneos` (device) wheel AS WELL as both simulator ones.** It configures pip for `iphoneos.arm64` + `iphonesimulator.arm64` + `iphonesimulator.x86_64` and needs a wheel for EACH — a partial local matrix fails with `No matching distribution found`. Build all three iOS slices first (for the recipe AND every `flet-lib*` host dep). CI never hits this because it dumps all of `dist/*.whl` into its find-links dir. (`flet build apk` needs only the one `--arch` slice — the asymmetry is iOS-only.) Long-standing gotcha; re-hit during the onnxruntime iOS spike. **Worse: a hand-patched wheel can still lose in serious_python's PER-ARCH native staging (`build/site-packages//`).** That pip run does read the inherited `PIP_FIND_LINKS` (flet 0.86.5 logs "Looking in links: …", and an unpublished package such as lz4 resolves from it), but during the opencv work a hand-patched `-9999` wheel in the find-links dir was used for the initial pip install while the staged PYTHON code (e.g. `cv2/__init__.py`, whichever slice it picks — often `iphoneos.arm64`) still comes from the published wheel. Net: you cannot validate a *hand-patched loader* on a local `ios-simulator` build; use a real `forge` build of all slices, or verify in CI (where the freshly-built slice wheels ARE used — this is why coolprop iOS passed in CI but a hand-patched opencv wouldn't locally). 10a. **Old local Xcode can't compile newer Flutter plugins** — e.g. Xcode 16.4 dies on `device_info_plus` 12.4.0 with `ARC Semantic Issue: No visible @interface for 'NSProcessInfo' declares the selector 'isiOSAppOnVision'` (a visionOS selector added in a newer SDK). This is a LOCAL toolchain gap, not your recipe (CI's Xcode 26.5 is fine). Pin the offending plugins older in the generated app pyproject before `flet build ios-simulator`: ```toml @@ -136,6 +160,26 @@ for i in $(seq 1 30); do grep EXIT "$DATA/Library/Caches/console.log" 2>/dev/nul ``` (`stage_recipe.sh` regenerates the pyproject, so append this AFTER staging.) +10b. **New local Xcode rejects flet's iOS deployment target.** Xcode 27 makes + `IPHONEOS_DEPLOYMENT_TARGET = 13.0` (flet's template from 0.86.5 through 1.0.3, still on + `main` as of 2026-10-03) a hard error: *"set to 13.0, but the range of supported deployment target + versions is 15.0 to 27.0.x"*. Plain `flet build` output hides it behind "Failed to build + iOS app" and a doctor dump; `-vv` shows the `error:` lines. Not your recipe, and CI's + Xcode 26 is unaffected. Workaround — flet has already staged everything, so patch the + generated project and re-run only the flutter step with the same env: + ```bash + cd build/flutter/ios + sed -i '' "s/platform :ios, '13.0'/platform :ios, '15.0'/" Podfile + # in Podfile's post_install target loop, before GCC_PREPROCESSOR_DEFINITIONS: + # config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '15.0' + sed -i '' 's/IPHONEOS_DEPLOYMENT_TARGET = 13.0;/IPHONEOS_DEPLOYMENT_TARGET = 15.0;/' Runner.xcodeproj/project.pbxproj + cd .. && SERIOUS_PYTHON_APP=$PWD/../python-app SERIOUS_PYTHON_SITE_PACKAGES=$PWD/../site-packages \ + SERIOUS_PYTHON_VERSION=3.14 SP_NATIVE_SET=$(cat ../.serious_python_spm_key) \ + flutter build ios --simulator --build-name 1.0.0 # VERSION = the Python flet built for + # → build/flutter/build/ios/iphonesimulator/.app + ``` + A fresh `flet build` regenerates the template, so repeat the patch after every one. + 11. **Two booted simulators make `simctl booted` ambiguous.** With more than one sim booted, `simctl install booted …` targets one device and your subsequent `get_app_container booted …` may query the OTHER — the app "isn't installed" / the container is empty despite a successful install. Use the explicit `$UDID` for every simctl call (as the loop above does); never rely on `booted` unless you've verified exactly one device is booted (`xcrun simctl list devices | grep -c Booted`). 12. **Verify the staged tests + the on-device test COUNT — staging can fail silently.** `stage_recipe.sh` wipes and re-stages `recipe_tests/`; if the invocation ever fails without you noticing (a scripted loop with a bad variable — zsh does NOT word-split unquoted `$VAR` like bash, so a `for r in $RECIPES`-style loop can pass the whole list as ONE argument), the PREVIOUS recipe's tests are still staged and run happily, reporting "N passed" for the wrong package. Two cheap checks after staging: `ls tests/recipe-tester/recipe_tests/` shows YOUR test files, and the "N passed" in console.log matches your recipe's test count. (Bit during the h5py→keras loop: the same 4 stale h5py tests "passed" three times.) **Stronger still — verify the built APK's CONTENTS, not just `recipe_tests/`:** a build that *fails* can leave a STALE `build/apk/recipe-tester.apk` that installs the wrong app entirely. `unzip -l build/apk/recipe-tester.apk` should show your recipe's test `.py` inside `app.zip` AND (for a native recipe) `lib//lib*.so` for its libs. Caught an opaque run that silently installed a stale pysodium APK and reported "2 passed" for the wrong package. When in doubt nuke `build/apk` too, not just `build/site-packages`. diff --git a/.claude/skills/new-mobile-recipe/SKILL.md b/.claude/skills/new-mobile-recipe/SKILL.md index 20cd0d95..115497cf 100644 --- a/.claude/skills/new-mobile-recipe/SKILL.md +++ b/.claude/skills/new-mobile-recipe/SKILL.md @@ -235,6 +235,18 @@ you can `export NDK_HOME=~/Library/Android/sdk/ndk/` and use that — f the differences between r27 / r27d / r28 are immaterial. CI builds against r27d; your local r27 should produce equivalent wheels. +### Trap: a truncated support tarball + +`download_support` in `setup.sh` returns early once the extracted +`downloads/support/python--mobile-forge-/support/` exists, otherwise skips the +download whenever the tarball exists, and never checks `tar`'s exit status. A shell killed +mid-`curl` leaves a partial tarball behind (2026-10-01: an iOS 3.14.6 tarball at 34 MB of +296 MB): `tar` reports `truncated gzip input`, and depending on how far it got, setup either +fails its support-path check or keeps a half-extracted tree on every later run. Compare a +cached tarball with the release (`curl -sIL | grep -i content-length` against +`stat -f%z`), and delete both the tarball and its extracted `downloads/support/…` directory +before re-running. On a slow link, fetch with parallel `curl -r` ranges and concatenate. + ### Note: Android sysconfigdata CI paths — self-healing, nothing to fix Historically the `python-android-mobile-forge-3.12.tar.gz` broke macOS local dev: CI-runner @@ -298,6 +310,9 @@ Open `tests/test_.py` and replace the placeholder with a real smoke test: - **Every test function has a docstring** — one line saying what behavior it proves. - Tests must be **network-free and deterministic** (fixed seeds, committed tiny assets) — they run on an emulator with no guarantees about connectivity. +- **Type every parameter where Python allows it** — tests, fixtures (`tmp_path: Path`), + helpers, and the example apps' functions and nested handlers alike. Only lambdas are + exempt (no annotation syntax); domain functions also get a return type. For ML/inference recipes, raise the bar from import-only to real compute: @@ -544,6 +559,29 @@ what's committed vs generated). `>>>>>>>>>> EXIT 0 <<<<<<<<<<` in console.log on both platforms = ready to ship. +### Audit the consumer README and example before the PR + +Every sentence in `recipes//README.md` and the example is a claim a bump can break, +and plausible ones are often wrong. An adversarial claim audit of lz4 (one skeptic per +claim group, each told to refute with source quotes and runs against the sdist, PyPI's +desktop wheel and the reference tools) turned up defects that a green CI could not: + +- **Upstream constants can overstate the library.** python-lz4 exports + `COMPRESSIONLEVEL_MAX = 16`, but the vendored liblz4 clamps HC at 12, so levels 13–16 + produce byte-identical output. Check a "max" against the vendored C, not the binding. +- **An example with several buttons and `page.run_thread` races itself.** The pool runs + workers concurrently, so a second tap mid-run overlapped the first on a shared file and + failed it every time. Disabling the controls in the handler is **not enough**: on an iOS + simulator two taps reached Python 11 ms apart, before the `disabled` patch reached + Flutter, and the race still fired. Guard the handler itself — + `if not busy.acquire(blocking=False): return`, release at the end of the worker — and + disable the controls for the visual cue (`recipes/lz4/examples/log-archive`). +- **"Stateful object, one per thread" understates it** when the binding mutates the native + context with the GIL released: sharing an `LZ4FrameCompressor` crashed the interpreter. +- **Prose that says "streams" needs code that streams** — one `write()` of the whole + buffer is not it, and neither is a test that never calls `write()` twice. +- **Run flake8 over the example** (`--config .flake8`); E741 slipped through review. + ### Wheel hygiene checklist (before commit) Unzip the wheel(s) from `dist/` and inspect every native binary: diff --git a/recipes/av/README.md b/recipes/av/README.md index 2a77b6aa..3fee0a42 100644 --- a/recipes/av/README.md +++ b/recipes/av/README.md @@ -133,13 +133,16 @@ control for playing the result. PyAV releases the GIL around demuxing, decoding and encoding, so this work genuinely runs in parallel with the UI. Hand it to [`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread) — a -half-second decode on the event-handler thread is a visibly frozen app. `run_thread` swallows -exceptions and does not carry an automatic update with it, so catch your own failures and -finish the handler with an explicit +half-second decode on the event-handler thread is a visibly frozen app. `run_thread` reports a +failure only in the log and does not carry an automatic update with it, so catch your own +failures and finish the worker with an explicit [`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update). A single container is not safe to use from two threads at once. Give each thread its own -`av.open(...)`, or serialise access with a `threading.Lock`. +`av.open(...)`, or serialise access with a `threading.Lock`. Two runs writing one output +file need a lock even with separate containers, and `run_thread` uses a pool, so two quick +taps on a button run concurrently. Take it in the click handler: disabling the button alone +does not stop a tap that is already in flight. ### What this build can encode diff --git a/recipes/av/examples/clip-roundtrip/README.md b/recipes/av/examples/clip-roundtrip/README.md index 23de0252..49e968df 100644 --- a/recipes/av/examples/clip-roundtrip/README.md +++ b/recipes/av/examples/clip-roundtrip/README.md @@ -21,7 +21,8 @@ What it demonstrates: - **Work off the UI thread** — the encode runs in [`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread), which needs an explicit [`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update) - at the end. + at the end. The handler also takes a non-blocking lock: every run rewrites the same clip, + and disabling the button does not stop a second tap that is already in flight. ## Try it diff --git a/recipes/av/examples/clip-roundtrip/src/clip.py b/recipes/av/examples/clip-roundtrip/src/clip.py index 00693b96..10622b75 100644 --- a/recipes/av/examples/clip-roundtrip/src/clip.py +++ b/recipes/av/examples/clip-roundtrip/src/clip.py @@ -1,7 +1,6 @@ -"""Writes a short MP4 with a video and an audio stream, reads it back to describe -what actually landed in the file, and pulls stills out of it as JPEG bytes an -`ft.Image` can display. -""" +"""Writes a short MP4 with a video and an audio stream, reads it back to describe what +actually landed in the file, and pulls stills out of it as JPEG bytes an `ft.Image` can +display.""" import io import math @@ -57,7 +56,7 @@ def _video_frame(index: int): return frame -def _audio_frames(stream): +def _audio_frames(stream: av.AudioStream): """A 440 Hz tone, resampled into whatever layout the AAC encoder asked for.""" resampler = av.AudioResampler( format=stream.format, layout=stream.layout, rate=SAMPLE_RATE @@ -129,7 +128,7 @@ def probe(path: str) -> list[tuple[str, str]]: return rows -def _jpeg(frame, width: int) -> bytes: +def _jpeg(frame: av.VideoFrame, width: int) -> bytes: """Re-encode one decoded frame as a JPEG, scaled to `width`.""" height = width * frame.height // frame.width scaled = frame.reformat(width=width, height=height, format="yuvj420p") diff --git a/recipes/av/examples/clip-roundtrip/src/main.py b/recipes/av/examples/clip-roundtrip/src/main.py index dee89231..a1efc0e5 100644 --- a/recipes/av/examples/clip-roundtrip/src/main.py +++ b/recipes/av/examples/clip-roundtrip/src/main.py @@ -1,8 +1,10 @@ +import threading + import flet as ft from clip import clip_path, library_versions, probe, thumbnails, write_clip -def still(label, jpeg): +def still(label: str, jpeg: bytes) -> ft.Column: """One filmstrip cell: a decoded frame above the timestamp it was taken at.""" return ft.Column( horizontal_alignment=ft.CrossAxisAlignment.CENTER, @@ -14,7 +16,7 @@ def still(label, jpeg): ) -def row(label, value): +def row(label: str, value: str) -> ft.Row: """One line of the probe readout: label on the left, what was read on the right.""" return ft.Row( alignment=ft.MainAxisAlignment.SPACE_BETWEEN, @@ -23,8 +25,15 @@ def row(label, value): def main(page: ft.Page): + busy = threading.Lock() + def run(): - """Lock the button, raise the spinner, and hand the work to a thread.""" + """Unless a run is in flight, disable the button, raise the spinner, and hand + the work to a thread.""" + # Every run rewrites the same clip. Disabling the button is not enough on + # its own: a second tap already in flight lands before the patch does. + if not busy.acquire(blocking=False): + return button.disabled = True spinner.visible = True page.update() @@ -33,8 +42,8 @@ def run(): def compute(): """Write the clip, probe it, extract stills, and update the page. - run_thread swallows exceptions and does not carry an automatic update - with it, so this catches its own failures and ends with page.update(). + run_thread reports a failure only in the log and does not carry an automatic + update with it, so this catches its own failures and ends with page.update(). """ try: path = clip_path() @@ -47,6 +56,7 @@ def compute(): button.disabled = False spinner.visible = False page.update() + busy.release() page.appbar = ft.AppBar(title=ft.Text("clip roundtrip"), center_title=True) page.add( diff --git a/recipes/lz4/README.md b/recipes/lz4/README.md new file mode 100644 index 00000000..0f48baaf --- /dev/null +++ b/recipes/lz4/README.md @@ -0,0 +1,158 @@ +# lz4 + +[`lz4`](https://python-lz4.readthedocs.io/en/stable/) binds Python to +[LZ4](https://lz4.org/), a lossless compressor built for speed rather than ratio. Its files +come out larger than zlib's, but it compresses several times faster and decompresses two to +three times faster, which on a phone means less CPU time and battery for every cached API +response, packed log or snapshot read back often. When size matters more than speed, the +standard library's [`zlib`](https://docs.python.org/3/library/zlib.html) needs no extra +wheel. + +## Install + +```toml +dependencies = [ + "flet", + "lz4", +] +``` + +## Examples + +See runnable Flet apps in [`examples/`](examples): + +- [`log-archive`](examples/log-archive) — writes a generated app log to an `.lz4` file at + three compression levels and reports ratio, timings and a verified round trip. + +## Usage in a Flet app + +Use [`lz4.frame`](https://python-lz4.readthedocs.io/en/stable/lz4.frame.html): its output is +the standard LZ4 format (see **Storage**), while +[`lz4.block`](https://python-lz4.readthedocs.io/en/stable/lz4.block.html) output is a bare +block behind python-lz4's own size prefix, which the `lz4` command-line tool cannot read. +For files, +[`lz4.frame.open`](https://python-lz4.readthedocs.io/en/stable/lz4.frame.html#lz4.frame.open) +returns a file object that compresses as you write to it, so data written in pieces never +has to sit in memory whole: + +```python +import os +import lz4.frame + +path = os.path.join(os.getenv("FLET_APP_STORAGE_DATA", "."), "events.lz4") +with lz4.frame.open(path, "wb") as f: + for chunk in chunks: + f.write(chunk) +``` + +`compression_level` picks the trade. The default (0) is LZ4's fast codec; 3 and up switch +to LZ4-HC, which compresses noticeably smaller and many times slower, while decompression +stays fast. HC tops out at 12 — liblz4 treats anything higher as 12, whatever +[`lz4.frame.COMPRESSIONLEVEL_MAX`](https://python-lz4.readthedocs.io/en/stable/lz4.frame.html#lz4.frame.COMPRESSIONLEVEL_MAX) +says. HC pays off for data written once and read often, +such as a bundled cache; the default suits anything written constantly, such as a log. The +[example](examples/log-archive) measures both on your device. + +In an app, run the work off the UI thread and put the result into a control: + +```python +status = ft.Text() + +def work(): + try: + packed = lz4.frame.compress(payload) + status.value = f"{len(payload):,} → {len(packed):,} bytes" + except Exception as e: + status.value = f"failed: {e!r}" + page.update() # a background thread needs this explicitly + +page.add(status, ft.Button("Compress", on_click=lambda _: page.run_thread(work))) +``` + +### Storage + +lz4 reads and writes nothing of its own — no config directory, no cache. Files you create +belong under +[`FLET_APP_STORAGE_DATA`](https://flet.dev/docs/reference/environment-variables/#flet_app_storage_data) +when they must survive, +[`FLET_APP_STORAGE_CACHE`](https://flet.dev/docs/reference/environment-variables/#flet_app_storage_cache) +for caches you can rebuild (the OS may purge it under storage pressure), or +[`FLET_APP_STORAGE_TEMP`](https://flet.dev/docs/reference/environment-variables/#flet_app_storage_temp) +for scratch that may vanish between launches. What `lz4.frame` writes is the standard LZ4 +frame format, so a file uploaded from the device decodes with the `lz4` command-line tool or +any other LZ4 implementation, with no Python on the server. + +### Threading + +[`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread) keeps +the UI responsive whatever lz4 call it runs, and the LZ4 calls themselves release the GIL, +so compressing (through any API) and +[`lz4.frame.decompress`](https://python-lz4.readthedocs.io/en/stable/lz4.frame.html#lz4.frame.decompress) +genuinely run in parallel with other threads. Reading through `lz4.frame.open` scales less +well, because Python code holding the GIL feeds the decompressor one +`io.DEFAULT_BUFFER_SIZE` at a time. That is 128 KiB on Python 3.14, which Flet bundles by +default, and readers calling `read(n)` with large chunks keep most of the speedup; it is +8 KiB on 3.12 and 3.13, where four readers finish no sooner than one after another. + +Catch exceptions inside the worker — `run_thread` does not surface them, and `lz4.frame` +reports corrupt input as a plain `RuntimeError`, or as `EOFError` for a file cut short +through `lz4.frame.open`, as when the app was killed mid-write — and finish with an explicit +[`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update). The one-shot +functions are safe from any number of threads. A compressor, decompressor or open `.lz4` +file is not: it has no lock, and lz4 changes it with the GIL released, so sharing one +between threads can crash the app. Give each thread its own. `run_thread` uses a pool, so +two quick taps run concurrently; guard work that must not overlap (two runs writing one +file, say) with a lock taken in the handler — disabling the button alone does not stop a +tap that is already in flight. + +### App size + +About 155–210 KB compressed and 280–650 KB unpacked per slice, all of it three small +compiled extensions; there are no data files. Small enough that it need not figure in which +ABIs you ship. + +## Build notes (maintainers) + +### Recipe shape + +Plain setuptools over a self-contained sdist, no patches. The package vendors liblz4 +(`lz4libs/`: the fast and HC codecs, the frame layer and xxHash), and each of its three +extensions statically compiles the parts it uses: `lz4._version` only the fast codec, +`lz4.block` adds HC, `lz4.frame` adds the frame layer and xxHash. PyPI's desktop wheels do +the same, so `flet run` and the device run the same liblz4. A separate shared `flet-liblz4` +would add a recipe and a load-time dependency for no consumer that needs one. + +### Upgrade hazards + +- **The version comes from setuptools_scm.** forge builds inside mobile-forge's own git + checkout; setuptools_scm does not search parent directories by default, so it falls back + to the sdist's `PKG-INFO`. Confirm the wheel filename and `lz4.__version__` still match + the recipe after a bump — a git-derived version here would be mobile-forge's. +- **`lz4.stream` is experimental upstream** and only built when `PYLZ4_EXPERIMENTAL` is + set, exactly as on PyPI. Leave it off unless upstream promotes it. +- **The HC ceiling is liblz4's, not python-lz4's.** If a bump moves the vendored liblz4 past + 1.9.x, re-check `LZ4HC_CLEVEL_MAX` in `lz4libs/lz4hc.h` before keeping the "tops out at + 12" sentence. + +### Re-verification checklist + +- **Wheel hygiene:** correct `Machine` per ABI, every Android `LOAD` segment aligned + `0x4000`, `DT_NEEDED` limited to bionic and `libpython` — a `liblz4.so` entry means the + vendored copy was bypassed. iOS `LC_BUILD_VERSION` platform 2 on device and 7 on the + simulators. +- **Interop:** a file written by the example decompresses with the desktop `lz4` CLI. Pull + the path the app shows from the iOS simulator's data container + (`xcrun simctl get_app_container com.flet.lz4-log-archive data`) or from a + `google_apis` emulator after `adb root`, then run `lz4 -d`. + `test_decodes_reference_cli_frame` covers the other direction. +- **Sizes:** re-measure from the wheels rather than scaling the figures above. + +### Coverage gaps + +The device tests cover the frame API with both checksums, the default, accelerated and HC +block modes with and without the size header, chunked compressor/decompressor objects, +`lz4.frame.open` written in pieces to device storage, and decoding a frame made by the +reference CLI. They do not cover dictionaries, `return_bytearray`, or concurrent use from +several threads. CI executes two of the six slices — Android `x86_64` on an emulator and +the arm64 iOS simulator; `arm64-v8a`, `armeabi-v7a`, the iOS device slice and the x86_64 +simulator slice are built and inspected but not run there. diff --git a/recipes/lz4/examples/log-archive/.gitignore b/recipes/lz4/examples/log-archive/.gitignore new file mode 100644 index 00000000..429a8307 --- /dev/null +++ b/recipes/lz4/examples/log-archive/.gitignore @@ -0,0 +1,7 @@ +.venv/ +.flet/ +build/ +__pycache__/ +.pytest_cache/ +.ruff_cache/ +uv.lock diff --git a/recipes/lz4/examples/log-archive/README.md b/recipes/lz4/examples/log-archive/README.md new file mode 100644 index 00000000..b2ae168b --- /dev/null +++ b/recipes/lz4/examples/log-archive/README.md @@ -0,0 +1,45 @@ +# lz4 log archive + +About 4 MB of app log, generated on the first tap. Each button writes it to an `.lz4` file +at one compression level, reads the file back, and reports the size on disk, the ratio, how +long each direction took, and whether every byte came back intact. The footer shows the +liblz4 version the wheel compiled in. + +What it demonstrates: + +- **The level trade, measured on the device.** `Fast` is LZ4's default codec; `HC 9` and + `HC 12` are LZ4-HC, and 12 is its ceiling. On a log like this HC files come out about 30% + smaller for roughly 15 to 150 times the write time, `HC 12` being the slow end, while + reading back stays fast at every level. +- **Writing and reading in pieces.** + [`lz4.frame.open`](https://python-lz4.readthedocs.io/en/stable/lz4.frame.html#lz4.frame.open) + returns a file object that compresses as it is written to, so the log goes to disk a + megabyte at a time and is checked back the same way. The file lands under + [`FLET_APP_STORAGE_TEMP`](https://flet.dev/docs/reference/environment-variables/#flet_app_storage_temp) + and is a standard LZ4 frame: the app shows its path, and once it is pulled off a + simulator or a rootable emulator, `lz4 -d` reads it. +- **Compute off the UI thread.** Each run happens in + [`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread) with the + level buttons locked and a spinner up, ending in the explicit + [`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update) a background + thread needs. Locking matters: every level writes the same file, and two overlapping runs + corrupt each other's read-back. Disabling the buttons is not enough on its own — a second + tap already in flight arrives before the disabled state does — so the handler also takes + a non-blocking lock and drops any tap that finds it held. + +The log is generated rather than bundled, so the example ships no asset. + +## Try it + +[Build](https://flet.dev/docs/publish/) the app, then install it on a device or emulator/simulator: + +```bash +# Android +uv run flet build apk + +# iOS +uv run flet build ipa + +# iOS-Simulator +uv run flet build ios-simulator +``` diff --git a/recipes/lz4/examples/log-archive/pyproject.toml b/recipes/lz4/examples/log-archive/pyproject.toml new file mode 100644 index 00000000..45a94c5b --- /dev/null +++ b/recipes/lz4/examples/log-archive/pyproject.toml @@ -0,0 +1,16 @@ +[project] +name = "lz4-log-archive" +version = "1.0.0" +description = "Writes a generated app log to an .lz4 file at three levels and verifies the round trip." +requires-python = ">=3.12" + +dependencies = [ + "flet==0.86.5", + "lz4==4.4.5", +] + +[dependency-groups] +dev = ["flet-cli", "flet-desktop", "flet-web"] + +[tool.flet.app] +path = "src" diff --git a/recipes/lz4/examples/log-archive/src/logs.py b/recipes/lz4/examples/log-archive/src/logs.py new file mode 100644 index 00000000..b68f1e3f --- /dev/null +++ b/recipes/lz4/examples/log-archive/src/logs.py @@ -0,0 +1,85 @@ +import functools +import os +import random +import tempfile +import time +from dataclasses import dataclass + +import lz4 +import lz4.frame + +# Below 3 is LZ4's fast codec; 3 and up is LZ4-HC, which liblz4 caps at 12. +LEVELS = {"Fast": 0, "HC 9": 9, "HC 12": 12} +CHUNK = 1 << 20 + + +@dataclass +class Archive: + raw: int + packed: int + write_s: float + read_s: float + intact: bool + path: str + + +def library_version() -> str: + """Version of the liblz4 compiled into the wheel.""" + return lz4.library_version_string() + + +@functools.cache +def sample_log(lines: int = 60_000, seed: int = 0) -> bytes: + """Build a deterministic app log of about 4 MB, standing in for a real one. + + Seeded so every run compresses the same bytes and the levels compare fairly; cached + so only the first tap pays for generating it. + """ + rng = random.Random(seed) + levels = ["DEBUG", "INFO", "INFO", "INFO", "WARN", "ERROR"] + routes = ["/feed", "/profile", "/search", "/upload", "/settings"] + stamp = 1_760_000_000.0 + out = [] + for i in range(lines): + stamp += rng.expovariate(20) + out.append( + f"{stamp:.3f} {rng.choice(levels):5} req={i:06d} " + f"route={rng.choice(routes)} status={rng.choice((200, 200, 304, 404, 500))} " + f"ms={rng.randint(3, 900)}\n" + ) + return "".join(out).encode() + + +def archive(data: bytes, level: int) -> Archive: + """Write `data` to an .lz4 file at `level` a megabyte at a time, then read it back + the same way, timing both ends and checking every byte. + + Chunked in both directions because that is how a log too large to hold twice would + be handled; lz4.frame.open compresses as it is written to. + """ + directory = os.getenv("FLET_APP_STORAGE_TEMP") or tempfile.gettempdir() + path = os.path.join(directory, "app-log.lz4") + view = memoryview(data) + + started = time.perf_counter() + with lz4.frame.open(path, "wb", compression_level=level) as f: + for i in range(0, len(data), CHUNK): + f.write(view[i : i + CHUNK]) + write_s = time.perf_counter() - started + + started = time.perf_counter() + intact, pos = True, 0 + with lz4.frame.open(path, "rb") as f: + while chunk := f.read(CHUNK): + intact &= view[pos : pos + len(chunk)] == chunk + pos += len(chunk) + read_s = time.perf_counter() - started + + return Archive( + len(data), + os.path.getsize(path), + write_s, + read_s, + intact and pos == len(data), + path, + ) diff --git a/recipes/lz4/examples/log-archive/src/main.py b/recipes/lz4/examples/log-archive/src/main.py new file mode 100644 index 00000000..2be2fb0e --- /dev/null +++ b/recipes/lz4/examples/log-archive/src/main.py @@ -0,0 +1,82 @@ +import threading + +import flet as ft +from logs import LEVELS, archive, library_version, sample_log + + +def main(page: ft.Page): + busy = threading.Lock() + + def run(name: str): + """Unless a run is in flight, disable the levels, raise the spinner, and hand + one archive run to a thread.""" + # Every level writes the same file. Disabling the buttons is not enough on + # its own: a second tap already in flight lands before the patch does. + if not busy.acquire(blocking=False): + return + levels.disabled = True + spinner.visible = True + page.update() + page.run_thread(lambda: work(name)) + + def work(name: str): + """Write, read back and verify at one level, then refill the report. + + run_thread reports a failure only in the log and does not carry an automatic + update with it, so this catches its own failures and ends with page.update(). + """ + try: + a = archive(sample_log(), LEVELS[name]) + headline.value = ( + f"{name}: {a.raw / 1e6:.1f} MB → {a.packed / 1e6:.2f} MB " + f"({a.raw / a.packed:.1f}x)" + ) + detail.value = ( + f"write {a.write_s * 1e3:.0f} ms ({a.raw / a.write_s / 1e6:.0f} MB/s)\n" + f"read {a.read_s * 1e3:.0f} ms ({a.raw / a.read_s / 1e6:.0f} MB/s)\n" + f"round trip {'intact' if a.intact else 'CORRUPTED'}" + ) + where.value = a.path + except Exception as exc: + headline.value = f"{name} failed" + detail.value = repr(exc) + where.value = "" + levels.disabled = False + spinner.visible = False + page.update() + busy.release() + + page.appbar = ft.AppBar(title=ft.Text("Archive a log"), center_title=True) + page.add( + ft.SafeArea( + expand=True, + content=ft.Column( + controls=[ + levels := ft.Row( + wrap=True, + controls=[ + ft.Button(name, on_click=lambda _, n=name: run(n)) + for name in LEVELS + ], + ), + ft.Row( + controls=[ + headline := ft.Text( + "Pick a level", size=18, weight=ft.FontWeight.BOLD + ), + spinner := ft.ProgressRing( + visible=False, width=18, height=18 + ), + ] + ), + detail := ft.Text(""), + where := ft.Text("", size=11, selectable=True), + ft.Divider(), + ft.Text(f"liblz4 {library_version()}", size=11), + ], + ), + ) + ) + + +ft.run(main) diff --git a/recipes/lz4/meta.yaml b/recipes/lz4/meta.yaml new file mode 100644 index 00000000..28400b19 --- /dev/null +++ b/recipes/lz4/meta.yaml @@ -0,0 +1,11 @@ +package: + name: lz4 + version: "4.4.5" + +build: + number: 1 + script_env: + # Always compile the liblz4 the sdist vendors. setup.py otherwise links any + # liblz4 >= 1.7.5 pkg-config can see -- none in the cross env today, but one + # appearing there would add a runtime library dependency nothing ships. + PYLZ4_USE_SYSTEM_LZ4: "0" diff --git a/recipes/lz4/tests/test_lz4.py b/recipes/lz4/tests/test_lz4.py new file mode 100644 index 00000000..d7d306bb --- /dev/null +++ b/recipes/lz4/tests/test_lz4.py @@ -0,0 +1,97 @@ +import random +from pathlib import Path + +# `lz4 -9 --content-size -BX` (the reference CLI, v1.10.0) over +# b"flet + lz4 on a phone\n" * 300: content size, block and content checksums. +CLI_FRAME = bytes.fromhex( + "04224d187c40c819000000000000ac3a000000ff07666c6574202b206c7a34206f6e2061" + "2070686f6e650a1600ffffffffffffffffffffffffffffffffffffffffffffffffffb350" + "686f6e650a07ddbf300000000019ef2271" +) +CLI_PAYLOAD = b"flet + lz4 on a phone\n" * 300 + + +def _payload() -> bytes: + """Half repetitive text, half seeded noise, so every codec path does real work.""" + text = b"".join(b"event %05d level=info msg=ok\n" % i for i in range(4000)) + return text + random.Random(0).randbytes(len(text)) + + +def test_frame_roundtrip_with_checksums(): + """A frame carrying size and both checksums round-trips -> lz4frame.c and the + vendored xxhash are compiled in, not just the block codec.""" + import lz4.frame + + data = _payload() + frame = lz4.frame.compress( + data, content_checksum=True, block_checksum=True, store_size=True + ) + info = lz4.frame.get_frame_info(frame) + assert info["content_checksum"] and info["block_checksum"] + assert info["content_size"] == len(data) + assert lz4.frame.decompress(frame) == data + + +def test_block_modes_roundtrip(): + """Default, accelerated and HC block modes round-trip, with and without the size + header -> covers both lz4.c and lz4hc.c in the _block extension.""" + import lz4.block + + data = _payload() + # "fast" only differs from "default" once acceleration is above 1. + for mode, extra in ( + ("default", {}), + ("fast", {"acceleration": 8}), + ("high_compression", {}), + ): + packed = lz4.block.compress(data, mode=mode, **extra) + assert lz4.block.decompress(packed) == data + + raw = lz4.block.compress(data, mode=mode, store_size=False, **extra) + assert lz4.block.decompress(raw, uncompressed_size=len(data)) == data + + +def test_incremental_matches_oneshot(): + """Chunked LZ4FrameCompressor/Decompressor output matches one-shot calls -> the + stateful frame context objects work on device.""" + import lz4.frame + + data = _payload() + step = 7000 + with lz4.frame.LZ4FrameCompressor() as compressor: + frame = compressor.begin(len(data)) + for i in range(0, len(data), step): + frame += compressor.compress(data[i : i + step]) + frame += compressor.flush() + + decompressor = lz4.frame.LZ4FrameDecompressor() + out = b"".join( + decompressor.decompress(frame[i : i + 512]) for i in range(0, len(frame), 512) + ) + assert decompressor.eof + assert out == data == lz4.frame.decompress(frame) + + +def test_frame_file_roundtrip(tmp_path: Path): + """lz4.frame.open writes a standard .lz4 file to device storage in pieces and reads + it back -> the file API streams, not only in-memory buffers.""" + import lz4.frame + + data = _payload() + path = tmp_path / "events.lz4" + with lz4.frame.open(path, "wb", compression_level=9) as f: + for i in range(0, len(data), 7000): + f.write(data[i : i + 7000]) + + assert path.read_bytes()[:4] == b"\x04\x22\x4d\x18" # LZ4 frame magic + assert path.stat().st_size < len(data) + with lz4.frame.open(path, "rb") as f: + assert f.read() == data + + +def test_decodes_reference_cli_frame(): + """A frame written by the reference lz4 CLI decodes on device -> the wheel + interoperates with lz4 outside Python, checksums verified.""" + import lz4.frame + + assert lz4.frame.decompress(CLI_FRAME) == CLI_PAYLOAD