Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .claude/skills/forge-error-catalogue/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,9 @@ instead of re-deriving it.
fix), **Rust crate with no `target_os="ios"` backend** (`mac_address`, cfg-gate it),
**`User for pypi.flet.dev:` → `EOFError` at build-tool/host-dep resolution** (the
index 401s, not the recipe — deterministic locally on a fresh cmake cross-venv,
transient/scattered in CI where a rerun clears it).
transient/scattered in CI where a rerun clears it). **A vendored lib built by
setup.py's OWN cmake call** (arg list hardcoded, so `CMAKE_ARGS` does nothing) →
green host-configured library, fix by patching in a `FORGE_CMAKE_ARGS` extend.
- **Runtime failures** (device/emulator/simulator) — **the Flet 0.86 Android
`sitepackages.zip` class** (its umbrella entry explains "why only now"):
`NotADirectoryError` on a bundled data file → **`extract_packages`** meta field;
Expand All @@ -104,7 +106,9 @@ instead of re-deriving it.
`libssl.so.3`/`libcrypto`/`libsqlite` not found, import-name errors, old version
loaded, **lazy_loader "non-existent stub" (serious_python strips `*.pyi`)**,
**hidden runtime deps** (keras→scipy; device-emulating venv method),
**insightface `root=` PermissionError**, and **iOS app crashes at launch with a
**insightface `root=` PermissionError**, **Android-only silent degradation when a lib
reads system config through a Java-only API** (c-ares gets no nameservers, seeds
`127.0.0.1:53`, every lookup fails while iOS is fine — configure it from Python), and **iOS app crashes at launch with a
0-byte `console.log` → `dyld: Library not loaded: @rpath/lib<X>.dylib` for a chain
of interdependent bundled dylibs (pyarrow, llama)** → **serious_python #223**:
reconcile framework install-ids + `@rpath` deps to the dotted-framework paths
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -803,6 +803,38 @@ run (sherpa's setup.py falls back to a bounded `make -j4`). **Related trap:**

---

### A vendored native lib is built by setup.py's OWN cmake call — green build, host-configured library

**Cause:** the sdist vendors a C library (`deps/<lib>/`) and `build_ext` shells out to
`cmake` with an argument list that is a **literal in setup.py**, then links the static
result via `extra_objects`. No CMake build backend is involved, so `CMAKE_ARGS` (which
only scikit-build-core reads) is ignored and the vendored lib configures for the build
host. Nothing errors: repeated arch (macOS arm64 host → `ios_arm64` target) links
without complaint, and the configure-time feature probes answer for macOS. The wheel is
green and wrong — either dead at first use, or quietly missing whatever the probes
turned off. pycares → c-ares 1.34.6.

**Fix:** patch the arg list to be extensible and fill it from a recipe `script_env` var:

```python
cmake_args.extend(shlex.split(os.environ.get('FORGE_CMAKE_ARGS', '')))
```

Append **after** upstream's own platform block — repeated `-D` on a cmake command line
is last-one-wins, so this overrides e.g. `-DCMAKE_OSX_DEPLOYMENT_TARGET=10.12` without
editing upstream's line. Then the usual lanes (`{NDK_ROOT}/build/cmake/android.toolchain.cmake`
+ `{ANDROID_ABI}` + `{ANDROID_API_LEVEL}`; `-DCMAKE_SYSTEM_NAME=iOS` + `{{ sdk }}` /
`{{ arch }}` / `{{ sdk_version }}`) and `requirements.build: [cmake]`.

**Confirm from the log, not the exit code:** `-- Check for working C compiler:` must name
the cross compiler (NDK clang / `arm64-apple-ios*-clang`), and the feature macros you
depend on must have resolved for the target — grep the generated config header under
`build/<py>/<pkg>/<ver>/build/temp.*/`. For pycares that is `HAVE___SYSTEM_PROPERTY_GET 1`
and `CARES_THREADS 1` on Android (`import pycares` raises `RuntimeError` outright if
`ares_threadsafety()` is false, which is a mercy — most losses of this class are silent).

---

### CMake-driving setup.py keys its build tool on a literal `"-G Ninja"` substring

**Cause:** sherpa-onnx's `cmake/cmake_extension.py` decides make-vs-ninja with
Expand Down Expand Up @@ -1797,6 +1829,35 @@ build a device-emulating desktop venv containing ONLY the wheel's declared deps

---

### Android only: a library that reads *system configuration* through a Java-only API silently degrades instead of failing

**Cause:** Android 8 removed most `net.*` / config system properties an app can read, and
the replacements live behind Java APIs (`ConnectivityManager`, `TelephonyManager`, …). A C
library that supports Android at all usually reaches them through JNI, and needs the app to
hand it a `JavaVM*` first. Nothing in the Flet/serious-python stack does that, and no pure
CPython wheel can — so the discovery returns nothing, and libraries of this class then
**seed a default rather than erroring**, leaving a working object that cannot do its job.

pycares/c-ares is the worked example: `ares_init_sysconfig_android()` needs
`ares_library_init_jvm()` + `ares_library_init_android()` (neither exposed by pycares),
falls back to the removed `net.dns1`…`net.dns8` properties, returns `ARES_EFILE` — and
`ares_init.c` swallows that and installs `127.0.0.1:53` as the sole nameserver. So
`pycares.Channel()` constructs fine, `channel.servers == ['127.0.0.1:53']`, and every
lookup fails with `DNSError: (11, 'Could not contact DNS servers')`. iOS is unaffected —
there c-ares `dlsym`s Apple's configd SPIs out of libSystem and gets the real resolvers.

**Tell:** the same code works on iOS and fails on Android, the error is a *connection*
failure rather than a configuration error, and the object's config property reads back a
loopback/default value.

**Fix:** there is no wheel-side fix — configure it explicitly from Python
(`pycares.Channel(servers=[...])` / `aiodns.DNSResolver(nameservers=[...])`), or read the
real values on the Java side with pyjnius and pass them in. Document it in the recipe
README; do not encode a public default in the wheel. A recipe test cannot assert any of
this (it differs per platform and network) — surface it with a consumer verify-app.

---

### insightface: `PermissionError: [Errno 13] … '/data/.insightface'` (FaceAnalysis init, Android)

**Cause:** `FaceAnalysis()` defaults its model root to `~/.insightface`, and in
Expand Down
34 changes: 33 additions & 1 deletion .claude/skills/new-mobile-recipe/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,43 @@ Match the package to one of these shapes. Each maps to a template in `templates/
| Native library, **ctypes-loaded (shared)** | A pure-Python wrapper `dlopen`s the lib at runtime via `ctypes` (pyzbar→libzbar, python-magic→libmagic) | `templates/meta-flet-lib.yaml` + `templates/build-flet-lib-shared.sh`; see Pattern H |
| Cython-accelerated pure-Python (poetry-core build script) | `build-backend = "poetry.core.masonry.api"` + `[tool.poetry.build] script` that cythonizes the runtime `.py` files themselves (zeroconf; the Home-Assistant-ecosystem idiom). Forge's PEP 517 path handles poetry-core unchanged | No template — copy `recipes/zeroconf/` (branch `zeroconf`): `script_env REQUIRE_CYTHON: "1"` + a fail-loud patch (upstream swallows compile errors → silent pure-py wheel), test asserts the modules are real extensions |
| C-ext that links a lib via a `*-config` tool | Compiled C-ext whose `setup.py` shells out to `pg_config`/`mysql_config`/… (psycopg2→libpq, mysqlclient→libmysqlclient) | A **static+PIC** `flet-lib*` (`build-flet-lib.sh` + `-fPIC`) shipping a config-shim, + consumer `script_env`/patch; see Pattern I |
| **setup.py that drives CMake itself** for a vendored native lib | An sdist that vendors a C library and builds it with its own `subprocess` CMake call inside `build_ext`, then links the static result via `extra_objects` (pycares→c-ares). Not scikit-build-core — the arg list is hardcoded in `setup.py` | No template — copy `recipes/pycares/`: one patch appends `shlex.split(os.environ['FORGE_CMAKE_ARGS'])` to the arg list, `requirements.build: [cmake]`; see "vendored-CMake" deep-dive below |
| CMake giant, **no sdist AND no setup.py/pyproject.toml** | Upstream's only wheel path is a host==target build script (onnxruntime's `ci_build/build.py`, TF's `build_pip_package_with_cmake.sh`) | No template — copy from `recipes/onnxruntime/` or `recipes/tflite-runtime/` (branches `machine/onnxruntime` / `machine/tflite-runtime`); see "PEP 517 shim" deep-dive below |
| **Prebuilt-repackage + host_build chain** | Upstream publishes official prebuilt mobile archives of the native lib AND the consumer's own cmake links + re-ships the `.so` (flet-libonnxruntime→sherpa-onnx) | `build.sh` repackager + consumer `requirements.host_build`; copy from `recipes/flet-libonnxruntime/` + `recipes/sherpa-onnx/` (branch `machine/sherpa-onnx`); see "prebuilt-repackage" deep-dive below |

If unsure, start with **minimal C-extension** and let the build tell you what's missing. Iterate up the table as failures surface.

### Shape deep-dive: setup.py that drives CMake for a vendored lib (pycares)

**When:** the sdist vendors a C library (`deps/<lib>/`) and its `build_ext` shells out to
`cmake` to build it, then links the resulting static lib into the extension with
`extra_objects`. It looks like a CMake recipe but no CMake build backend is involved, so
`CMAKE_ARGS` (which only scikit-build-core reads) does nothing — the argument list is a
literal in `setup.py`, configured for the build host.

The whole recipe is one patch that makes that list extensible, plus the env var to fill it:

```python
cmake_args.extend(shlex.split(os.environ.get('FORGE_CMAKE_ARGS', '')))
```

Append it **after** upstream's own platform block so the recipe's args win — repeated `-D`
on a cmake command line is last-one-wins, which is how `-DCMAKE_OSX_DEPLOYMENT_TARGET`
gets overridden without touching upstream's line. Then the usual per-SDK `script_env`
lanes (`{NDK_ROOT}/build/cmake/android.toolchain.cmake` + `{ANDROID_ABI}` +
`{ANDROID_API_LEVEL}` on Android; `-DCMAKE_SYSTEM_NAME=iOS` + `{{ sdk }}` / `{{ arch }}` /
`{{ sdk_version }}` on iOS), and `requirements.build: [cmake]`.

**Why this shape is worth naming: getting it wrong produces a green wheel.** A host-configured
vendored lib still *links* whenever host and target arch agree (macOS arm64 objects go into an
`ios_arm64` extension without complaint), and the resulting wheel then fails on device — or
worse, works while quietly missing a feature the configure step probed for. Verify from the
build log, not the exit code: `Check for working C compiler:` must name the cross compiler,
and the feature macros that matter must have resolved for the target (for pycares:
`HAVE___SYSTEM_PROPERTY_GET` on Android, `Found Threads: TRUE` on both — a threadless c-ares
makes `import pycares` raise outright). Grep the generated `ares_config.h`-equivalent in
`build/<py>/<pkg>/<ver>/build/temp.*/` when in doubt.

### Shape deep-dive: PEP 517 shim for no-sdist CMake giants (onnxruntime, tflite-runtime)

**When:** no sdist and no usable `setup.py`/`pyproject.toml` at the source root — upstream's supported wheel path is a host==target script. Forge's `PythonPackageBuilder` is PEP 517-only, so the recipe's `mobile.patch` **ADDs** the PEP 517 entry instead of fighting the script: a `pyproject.toml` with `build-backend = "forge_<pkg>_backend"` + `backend-path = ["_forge"]`, plus a new `_forge/forge_<pkg>_backend.py` that wraps `setuptools.build_meta` (`from setuptools.build_meta import *`, then override the hooks). `source.url` points at the GitHub tag tarball.
Expand All @@ -95,7 +127,7 @@ The shim's load-bearing rules (each one bought with a failed build):
- **Stage the python package fresh on every hook** (overlay / rm+copy from the current slice's build dir into the source root) so the current slice always wins.
- Prefer **`-D<pkg>_BUILD_SHARED_LIB=OFF` → one statically-linked pybind module**: no ctypes/dylib gate, and it is exactly the wheel shape that works on BOTH platforms today (onnxruntime's Android design turned out to BE the iOS wheel shape; pywhispercpp/ncnn are the same shape).
- CMake args ride in a recipe `script_env` var (`FORGE_CMAKE_ARGS`) that the shim `shlex.split`s. **Multi-token `-D` values cannot ride inside it** — give linker-flag strings their own env var and let the shim assemble the single `-D` argument (tflite's `FORGE_SHARED_LINKER_FLAGS: -Wl,-z,max-page-size=16384 -L{HOST_PYTHON_HOME}/lib -lpython{py_version_short}`).
- **Trap — `setup.py` platform predicates:** under the crossenv `platform.system()` returns `"Android"` / `"iOS"`, which may match NO upstream branch → the libs list stays unset and the wheel silently ships **without the pybind `.so`** (onnxruntime extends upstream's predicate to `("Linux", "AIX", "Android", "iOS")`; the Darwin branch stays intact for real macOS builds).
- **Trap — `setup.py` platform predicates:** under the crossenv `platform.system()` returns `"Android"` / `"iOS"`, which may match NO upstream branch → the libs list stays unset and the wheel silently ships **without the pybind `.so`** (onnxruntime extends upstream's predicate to `("Linux", "AIX", "Android", "iOS")`; the Darwin branch stays intact for real macOS builds). `sys.platform` is the same story one layer down — crossenv sets it to `"android"` / `"ios"` (from the sysconfigdata's `_PYTHON_HOST_PLATFORM`), so `sys.platform.startswith('linux')` and `== 'darwin'` are both False. **Matching nothing is sometimes exactly right**: it is what keeps `-lrt` (no librt in bionic) and a macOS deployment target out of the pycares build, which is why that recipe needs no platform patch at all. Read the branches before assuming you must extend them.

**Real examples:** `machine/onnxruntime:recipes/onnxruntime/` and `machine/tflite-runtime:recipes/tflite-runtime/` — read via `git show <branch>:<path>` if not on those branches. Both are ONE `mobile.patch` (description as text preamble above the first `---` header, per repo convention).

Expand Down
Loading
Loading