Skip to content

recipe: lz4 4.4.5 - #124

Merged
ndonkoHenri merged 9 commits into
mainfrom
lz4
Oct 4, 2026
Merged

ndonkoHenri merged 9 commits into
mainfrom
lz4

Conversation

@ndonkoHenri

Copy link
Copy Markdown

Adds a recipe for lz4 4.4.5 — Python bindings to LZ4, the lossless compressor built for speed rather than ratio. Requested in flet#6908.

On a phone the cost of compression is CPU time and battery, which is exactly what LZ4 trades ratio away to save: its files are larger than zlib's, but it compresses several times faster and decompresses two to three times faster.

Recipe shape

Plain setuptools over the PyPI sdist, no patches. The sdist vendors liblz4 (lz4libs/: the fast and HC codecs, the frame layer and xxHash) and each of the three extensions statically compiles the parts it uses, exactly as PyPI's desktop wheels do — so flet run and the device run the same liblz4.

One script_env setting: PYLZ4_USE_SYSTEM_LZ4: "0". setup.py otherwise links any liblz4 >= 1.7.5 that pkg-config can see. Forge's PKG_CONFIG_LIBDIR hides the build host's today, but a liblz4 appearing in the cross env later would silently turn the self-contained wheel into one with a runtime dependency nothing ships.

Validation

  • CI 6/6 jobs green across 3.12 / 3.13 / 3.14 × android / iOS with mobile_test_pythons=ALL, twice (fork runs 36916076140 and 37141952995, the latter on this recipe and test tree). On-device 5 passed / EXIT 0 on every leg, each confirmed to run its own interpreter.
  • 18 wheels inspected: right Machine per ABI, every Android LOAD segment aligned 0x4000, DT_NEEDED limited to bionic and libpython (no liblz4.so), iOS LC_BUILD_VERSION platform 2 on device / 7 on the simulators, lz4.__version__ 4.4.5 from PKG-INFO (setuptools_scm does not pick up mobile-forge's own git).
  • The log-archive example builds and runs on the iOS Simulator and on an arm64-v8a Android emulator — the slice real phones use, which CI never executes. Files it writes decode byte-identical with the reference lz4 CLI on both; test_decodes_reference_cli_frame covers the other direction on device.
  • Before the PR, an adversarial audit of every claim in the README, example and tests (one skeptic per claim group, each running the code against the sdist, PyPI's wheel and the CLI). It caught things a green CI could not, all fixed here: liblz4 clamps LZ4-HC at level 12 although python-lz4 exports COMPRESSIONLEVEL_MAX = 16; sharing one LZ4FrameCompressor between threads can crash the interpreter; the test's "fast" block mode was byte-identical to "default" until given an acceleration; and the example raced itself on a double tap. A second review of everything after that audit corrected one more claim: readers through lz4.frame.open do run in parallel on Python 3.14 (its read buffer grew from 8 KiB to 128 KiB), just not on 3.12/3.13.

The double-tap race (also fixed in the av example)

Disabling a button in its click handler does not stop a second tap that is already in flight: on the iOS Simulator two taps reached Python 3–48 ms apart, before the disabled patch reached Flutter, and both started a page.run_thread worker. In the lz4 example the two runs share one file, and the first run's read-back failed every time. Both handlers now take a non-blocking threading.Lock and drop a tap that finds it held; verified on device that a double tap produces exactly one run.

recipes/av/examples/clip-roundtrip (#122) had the same shape. There the overlap showed up as the button and spinner resetting while a second encode was still writing the clip — no corrupt output was observed — so it gets the same lock, and the av README's threading section gets the same advice.

CI on this PR rebuilds av as well, only because its example and README changed: the changed-recipe filter covers all of recipes/**. The av recipe and its published wheels are untouched.

Changes

  • recipes/lz4/ — meta.yaml, 5 on-device tests, README.md, and the log-archive example.
  • recipes/av/ — the example's tap guard and one paragraph in each README.
  • .claude/skills/ — running the consumer example on CI's wheels with no local forge env; Xcode 27 rejecting flet's iOS deployment target (13.0) and the local workaround; the emulator's missing-system-image recovery; gh job logs needing --allow-escape-sequences; truncated support tarballs being reused silently by setup.sh; and an "audit the consumer README before the PR" checklist from the findings above.

Consumer notes

lz4 gives a Flet app fast lossless compression for caches, logs and uploads; what it writes is the standard LZ4 frame format, so a server can decode it without Python. The things that are different on a phone — where files belong in Flet's storage, which calls really run in parallel under run_thread, why a compressor must not be shared between threads, and the HC level ceiling — are in the recipe README, with a runnable example.

Plain setuptools over the PyPI sdist, which vendors liblz4 (fast + HC
codecs, frame layer, xxHash). No patches. PYLZ4_USE_SYSTEM_LZ4=0 keeps
setup.py's pkg-config probe from swapping the vendored copy for an
external liblz4 the wheel wouldn't ship.

Tests cover the frame API with both checksums, every block mode, the
chunked compressor/decompressor, lz4.frame.open on device storage, and
decoding a frame written by the reference lz4 CLI.

Adds the consumer README and a log-archive example app.

Requested in flet-dev/flet#6908.
- Docs: HC caps at 12 in liblz4 (13-16 behave as 12); sharing a
  compressor/decompressor/file across threads can crash; lz4.frame.open
  reads are GIL-bound; three storage dirs; App size section; CI runs only
  two of six slices.
- Example: lock the level buttons during a run (overlapping taps corrupted
  the shared file), write/read in 1 MiB chunks, HC 16 -> HC 12.
- Tests: "fast" block mode with acceleration 8 (at 1 it equals "default");
  the file test now writes in pieces.
Disabling the level buttons left a window: on the iOS simulator two taps
reached Python 11 ms apart, before the disabled patch landed, and the
overlapping runs still corrupted the shared file. The handler now takes a
non-blocking lock. Docs carry the same advice, and the example's timing
claims now match device measurements (iOS sim + arm64-v8a emulator).
- local-recipe-testing: run the consumer example on CI's wheels (no local
  forge env); Xcode 27 rejects flet's iOS deployment target 13.0 and how
  to work around it; emulator ANDROID_SDK_ROOT + missing system image.
- forge-ci: gh job logs need --allow-escape-sequences; confirm per-leg
  Python after mobile_test_pythons=ALL.
- new-mobile-recipe: truncated support tarballs are reused silently;
  audit the consumer README/example before the PR (upstream constants,
  run_thread races, thread-shared native state, streaming claims, flake8).
The clip-roundtrip example disabled its button in the click handler, but a
second tap already in flight reaches Python before the disabled patch
lands: on an iOS simulator two taps arrived 27 ms apart and started two
overlapping runs rewriting the same clip, with the button and spinner
reset while the second was still going. The handler now takes a
non-blocking lock (one run per double tap, verified on the iOS simulator
and an arm64-v8a emulator). README threading advice extended to match.
- README: lz4.frame.open readers do scale on Python 3.14 (128 KiB
  DEFAULT_BUFFER_SIZE), only not on 3.12/3.13; a truncated file read
  through it raises EOFError; lz4.block wording; link API names; drop a
  dangling intro sentence.
- Examples (lz4 + av): restore main() docstrings, describe the in-flight
  guard and run_thread's log-only failures accurately, black/docformatter.
- av README: clearer lock advice; run_thread reports failures in the log.
- local-recipe-testing: the emulator's ANDROID_SDK_ROOT messages both mean
  a missing system image, and the SDK's own sdkmanager must install it
  (Homebrew's uses another root); per-arch staging does read
  PIP_FIND_LINKS; 10b covers flet 0.86.5-1.0.3 and passes SP_NATIVE_SET;
  CI-wheels shortcut downloads to /tmp, no padding.
- forge-ci: the job-log snippet carries --allow-escape-sequences; the
  per-leg check greps pip's joined (wrapped) install line.
- new-mobile-recipe: truncated-tarball trap moved after the NDK fix and
  corrected (setup.sh also skips re-extraction once support/ exists).
@ndonkoHenri ndonkoHenri changed the title recipe: lz4 4.4.5 + av example tap guard recipe: lz4 4.4.5 Oct 4, 2026
Annotate all parameters in the lz4 and av examples and the lz4 tests
(tmp_path: Path; domain functions also get return types). Lambdas are the
only exemption. The rule joins the new-mobile-recipe test conventions.
av's clip.py also gets the docformatter rewrap it already failed on main.
@ndonkoHenri
ndonkoHenri merged commit 88290ff into main Oct 4, 2026
8 of 20 checks passed
@ndonkoHenri
ndonkoHenri deleted the lz4 branch October 4, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant