Wake-word plan (docs, M0 decisions) and M2: oracles for the learned suppressor - #45
Merged
Conversation
Background research and a staged proposal for adding wake-word detection to the family. Neither is a commitment to build it; both are the record of the question being asked properly. docs/wake-word-briefing.md surveys how keyword spotters actually work — the cascade, the feature front end, the model lineage from keyword/filler HMMs through Deep KWS to DS-CNN and dilated TDNN, the posterior-smoothing decision stage — and the free corpus landscape, with licences flagged where they block commercial use. docs/wake-word-plan.md proposes reusing what MuTap already has rather than importing a runtime: promote the dense/GRU arithmetic out of nn_suppressor.h into a shared tap::dsp::nn, add a mel/PCEN front end to DspTap, extend tools/ml to a second task, and ship one external. Milestones M0-M7 with pass criteria in the HANDOFF.md manner, and the evaluation harness deliberately sequenced before the model — false-accepts-per-hour is the metric that decides whether any of this works, and a harness built after the model is a harness built to flatter it. Every number in both documents is a target or a published corpus figure, never a measured result. Open decisions (repository home, wake phrase) are recorded, not assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ZmNVvP9oKtEkAZCfdQGy
Sixteen-agent audit of docs/wake-word-plan.md rev 1: seven adversarial reviewers (front end, codebase fit, architecture, evaluation, licensing, embedded, scope), a skeptic per lens re-checking every finding against the checkouts, a completeness critic and a cross-lens judge. Verdict: the design direction survives; the claim that MuTap has already solved the infrastructure for the learned path does not. nn_suppressor is instantiated only in double everywhere, is absent from the M55 on-target filter and the icount scenarios, and test_parity.py is a hand-run double-only random-weights check outside CI. Host sample-rate policy and the non-power-of-two 10 ms hop are undecided. The report ranks twelve root issues, records seven findings the judge discounted, and lists the amendments a rev 2 of the plan should carry. HANDOFF.md's wake-word entry points at the audit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Rewrites docs/wake-word-plan.md from the audit's fourteen-item list. The bet is restated as patterns and rigs inherited, oracles to be built. A new M2 builds the float, on-target, instruction-count and CI-parity verification for the learned suppressor before M3 promotes its kernels; rev 1's M2-M7 become M3-M8. M0 grows from two decisions to five (repository, host-rate policy, release shape, phrase, TTS voice lineage) plus a named compute budget, with recommendations in section 9: MuTap, a fixed internal 16 kHz with the external decimating, and runtime-first. Section 5 adds the contract points the audit found missing (FFT size and padding, frame alignment, bin-0, pre-emphasis, PCEN state, detection latency, streaming-state policy, accumulator precision) and an ownership rule: log_mel.h owns formulas, the MUKW weights carry every tunable. Section 4 corrects the TTS, SLR28 and openWakeWord rows and adds attribution delivery. M4 specifies speaker-disjoint splits across MSWC and Common Voice, a manifest schema, a feature store outside git, a hold-out specification and the label-form decision. M5's pass becomes a planted-event oracle that can fail and runs through the C ABI. Section 7 describes icount.py as a drift gate and adds absolute cost ceilings as targets. "No CMSIS-NN" is replaced by the fft.h backend rule. Factual errors in the asset table (Catch2, prepare() rule, sample_traits role) are fixed. The audit and the HANDOFF entry note that rev 2 landed and how the milestone numbers shifted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Max runs its DSP at whatever rate the audio driver offers, so a patch can run at 16 kHz outright where the interface supports it, and poly~ @down N gives a 16 kHz subpatch at 32/48/96 kHz otherwise. The spotter therefore runs at one internal rate and refuses others; the DspTap decimator is dropped from M1 and deferred until an embedded target with a fixed ADC clock needs one. 44.1 kHz has no integer path and stays unsupported in the first release. M8's help patcher and live-microphone pass now cover both the native 16 kHz and the poly~-wrapped 48 kHz cases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Corrects the previous revision on two facts: poly~ resamples by powers of two only, so it offers no 48 -> 16 kHz path; and RatioTap is 44.1 <-> 48 kHz by charter, with SampleRateTap near-unity async, so neither covers 48 -> 16 either. The spotter stays 16 kHz-only in kws.h; conversion becomes an @Resample option on mutap.wake~ in the Max layer, backed by a new integer-ratio decimator (2, 3, 6) that returns to M1 in RatioTap's design pattern, with 44.1 kHz handled by composing RatioTap's 44.1 -> 48 in front of the 3:1 stage. The decimator's home (DspTap primitive, recommended, or a RatioTap sibling) is an M0 sub-decision. RatioTap joins the existing-assets table and the diagram. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Names the Raspberry Pi Pico 2 W (RP2350) as the embedded M33 target. Its Cortex-M33 has a single-precision FPU, so float32 is the profile there and the plan's earlier framing of M33 parts as "without usable float" is corrected to "without double"; the nn contract now forbids double anywhere on the spotter's hot path. M2 ports RatioTap's existing M33 QEMU rig (mps2-an505 toolchain, linker script, startup, CI job, ratchet) into MuTap and seeds baselines on all three emulated targets; M7 adds an out-of-tree Pico SDK example on the board itself with the hardware cycle count recorded beside the QEMU figure. Section 7 gains a targets table and an M33 column on the ceilings. Closes the documentation gap the runtime-first release exposed: a user guide to training a phrase (book chapter plus tools/ml/kws README), drafted at M6 and finished at M8, whose commands are a script CI runs on a toy corpus. Section 6 gains a table of every documentation surface with its milestone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
M7 gains a pico2w job beside the QEMU legs: the arm-none-eabi-gcc the M33/M55 legs already install, a Pico SDK pinned by tag and commit, PICO_BOARD=pico2_w, the example built, its UF2 uploaded as a workflow artifact, and the flash and RAM footprint from the linker map asserted against the section 7 ceilings. QEMU has no RP2350 model, so detection on the board stays a bench step of the pass criterion. The targets table, CI gates and documentation table say so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Music joins the evaluation negatives (MUSAN's music partition and permissively licensed Free Music Archive tracks), with the DET report splitting false accepts per hour on speech and on music, since a Max user's room has music in it. The Pico 2 W example specifies an I2S MEMS microphone, with PDM demodulation named as a budgeted follow-up rather than a free alternative, and its hardware pass becomes a bench protocol: loudspeaker playback of the hold-out set and an hour of negatives at a stated distance and level, scored by the M5 harness, numbers committed. The hold-out set is recorded through both the host interface and the Pico microphone path. The MUKW payload carries the front-end contract version and the external refuses a mismatch. The exporter also emits a C header of the weights image for flash-resident targets. Model loading in the Max external happens off the audio thread. The header gains a privacy statement and names the microphone path and the absence of a low-power tier among its limits. The training guide's toy CI run names its dependencies and time budget. The Wi-Fi follow-up notes PIO sharing with the wireless chip. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Records the M0 decisions in the plan and HANDOFF. Repository: MuTap, charter widened. Host rate: fixed internal 16 kHz in kws.h, conversion as an @Resample option on the Max external backed by a DspTap decimate.h, composed with RatioTap for 44.1 kHz. Release shape: runtime-first, no bundled phrase. Development phrase: marvin from Speech Commands for M5 bring-up, a synthesized phrase from M4, never shipped. Compute: an Apple Silicon Mac via --device mps. TTS voice lineage verified from the Piper per-voice model cards. Most English Piper voices, libritts_r and vctk included, are fine-tuned from the Lessac voice and inherit its research-only lineage; four are trained from scratch on permissive data and are adopted: en_US-libritts-high (LibriTTS, CC BY 4.0, 904 speakers), en_US-kristin-medium and en_GB-cori-high (LibriVox, public domain), and en_US-john-medium (from Kristin). The sample generator's bundled LibriTTS-R generator is excluded until its base voice is verified. The licensing map's Piper row carries the result; section 9 is retitled as the record of arguments. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Records M1 as built on DspTap's branch, with the measured tolerances that the plan required as committed numbers, the searched tap counts, and one deviation from the plan text: the numpy reference restatement lives in DspTap (tools/reference/make_frontend_reference.py) rather than MuTap's tools/ml, so the family has one numpy copy of the formulas and M4's feature module imports it through the submodule. Also notes that PCEN's steady state keeps a deliberate residual level dependence, which the test pins to the closed form. HANDOFF points at M2 next. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
The wake-word plan's M2 — build the verification the learned path should already have, before M3 refactors its kernels into DspTap. - nn_suppressor tests typed over float and double, with a cross-precision pin (float vs double -129.4 dB measured, pinned at -120) and a learned chain gate in test_float32.cpp (-91.4 dB measured, pinned at -85). - Numeric contract written into the nn_suppressor.h docstring as observed: every dot product accumulates in Sample; nothing in the float profile touches double. - Contract defect found by the oracles and fixed on both sides: the ERB top edge round trip erb_inv(erb_rate(fs/2)) lands 2.2e-11 below fs/2 in libm at 48 kHz, so the strict band test dropped the Nyquist bin and the C++ suppressor notched bin N/2 while the numpy reference did not (48 kHz parity disagreed by 3e-2 / 8e-3). The top edge is now fs/2 exactly and the Nyquist bin belongs to the last band in nn_suppressor.h and features.py alike. - tools/ml/test_parity.py promoted to a CI job (nn-parity): both profiles (nn_infer --float), random weights at 16 k and 48 k plus the exported suppressor_v2_48k.munn; measured double <= 2.9e-8, float <= 2.9e-7, pinned at 1e-6. - Cortex-M33 leg ported from RatioTap: cmake/arm-cortex-m33-mps2.cmake, platform/mps2_an505.ld, shared armv8m_startup.c, Ooura float32 FFT (no MVE), cortex-m33-qemu CI job, m33 icount target and ratchet step. This is the Raspberry Pi Pico 2 W (RP2350) class of core. - The float nn_suppressor suite, the cross-precision pin and the chain gate added to the on-target filter on M33 and M55 (Hexagon runs everything). - bench/icount layer 4: nn_suppressor at both trained geometries with xorshift weights; m55 and m33 baselines seeded locally (the local ratchet reproduces every committed m55 baseline exactly). Hexagon's two entries are seeded from the first CI log per bench/README.md. - docs/wake-word-plan.md M2 record with the measured numbers; HANDOFF.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
…o unused helpers The two learned-path scenarios had no hexagon entry, so the first ratchet run reported them as NO BASELINE (nn_suppressor_16k 465537141, nn_suppressor_48k 154160506, from the CI log of that run, per bench/README.md). The same run reproduced every m55 and m33 baseline at +0.00 %. Each icount binary is one scenario, so preset() is unreferenced in the learned-path binaries and nn_weights() in the others; both are now [[maybe_unused]] rather than warning on every cross build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
The first full run of the Cortex-M33 selection timed out at ctest's 1800 s: kalman_loop_test/0.PemAddsStableGainOnTonal alone took 1030 s under qemu mps2-an505 (84 s on mps3-an547). The cost is not the canceller — the closed-loop and echo harnesses simulate the room in double on purpose (tests/support/closed_loop.h convolves the feedback path in double per sample, the MSG bisection measures in double) and the speech predictor's pitch search accumulates in double — which is hardware on the M55 and software on the M33. The M33 toolchain now sets MUTAP_ON_TARGET_SOFT_FP64; tests/CMakeLists.txt forwards it as a compile definition and bare_metal_main.cpp drops the four long float PEM scenarios (kalman_loop_test/0, pem_afc_test/0, burst_test/0, aec_test/0) under it. Those stay on the M55 and Hexagon legs and every host. The M55 selection is unchanged. Measured: 47 tests, 69 s, all nine learned-path tests included; the wake-word plan's M2 record carries the finding as the first concrete cost figure for double on the RP2350 class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G
tap
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Two things on one branch, in commit order.
Docs.
docs/wake-word-briefing.md(the background survey),docs/wake-word-plan.md(the implementation proposal, rev 2, with dated "Done" records for M1 and M2) anddocs/wake-word-audit.md(a sixteen-agent adversarial audit of rev 1, every finding verified against the checkouts). HANDOFF.md's wake-word entry carries the M0 decisions and the M1/M2 results.M2 — oracles for the learned path, built on the unmodified suppressor before M3 refactors its kernels into DspTap:
tests/test_nn_suppressor.cpptyped over float and double, with a float-vs-double pin (−129.4 dB measured, pinned at −120) and a learned-chain gate intests/test_float32.cpp(−91.4 dB measured, pinned at −85).nn_suppressor.has observed: every dot product accumulates inSample; nothing in the float profile touches double.tools/ml/test_parity.pypromoted to a CI job (nn-parity): both profiles (nn_infer --float), random weights at 16 k and 48 k plus the exportedsuppressor_v2_48k.munn. Measured double ≤ 2.9e-8, float ≤ 2.9e-7, pinned at 1e-6.cmake/arm-cortex-m33-mps2.cmake,platform/mps2_an505.ld, sharedarmv8m_startup.c, Ooura float32 FFT since there is no MVE,cortex-m33-qemuCI job,m33icount target and ratchet step). This is the Raspberry Pi Pico 2 W (RP2350) class of core.nn_suppressorsuite, the cross-precision pin and the chain gate added to the on-target selection on M33 and M55 (Hexagon runs everything).bench/icountlayer 4:nn_suppressorat both trained geometries with xorshift weights; baselines on m55, m33 and hexagon.Why
The audit found the design direction sound but the rev 1 claim that MuTap had "already solved the infrastructure" wrong for the learned path: the suppressor was double-only everywhere, never on-target, never instruction-counted, and parity was a hand-run script. M2 builds those oracles first, and they earned their keep immediately (see below).
Verification
erb_inv(erb_rate(fs/2))lands 2.2e-11 below fs/2 in libm at 48 kHz (above at 16 kHz; above at both in numpy), so the strictf < hiband test dropped the Nyquist bin: the C++ suppressor notched bin N/2 at 48 kHz while the numpy reference did not. Before the fix the 48 kHz parity cases disagreed by 3e-2 (random) / 8e-3 (v2 model) in both profiles and the shipping-geometry test read −27.9 dB on unit gains. The top edge is now fs/2 exactly and the Nyquist bin belongs to the last band, innn_suppressor.handfeatures.pyalike.NO BASELINEreport perbench/README.md.cortex-m33-qemuandnn-parityjobs.MUTAP_ON_TARGET_SOFT_FP64andbare_metal_main.cppdrops those four scenarios there (47 tests, about 70 s); they stay on the M55 and Hexagon legs and every host. The M55 selection is unchanged.Measured per-hop cost of the existing GRU suppressor, whole-binary count over hops processed (setup included, so an upper bound): m55 ≈ 383 k instructions/hop at 48 kHz and ≈ 271 k at 16 kHz; m33 ≈ 711 k and ≈ 491 k. Against the plan's 150 k/hop wake-word ceiling, that is the number the M6 spotter has to beat by 2–5×.
Notes for the reviewer
main.🤖 Generated with Claude Code
https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G