Spec: C# probe re-implementing Test 2 (wakeword) on the Pi

2026-06-12 10:55:13 +00:00
parent bfd8a05734
commit 815389ecc2
1 changed files with 263 additions and 0 deletions
@@ -0,0 +1,263 @@
+# Test 2 wakeword C# probe — design
+
+Date: 2026-06-12
+Branch: `fresh-start`
+Status: approved (pending written-spec review)
+
+## Purpose
+
+Prove that openwakeword's stock "alexa" detector can be driven from .NET on the Pi, by porting `openwakeword/model.py`'s `predict()` pipeline (three ONNX models chained: mel-spectrogram → speech embedding → keyword classifier) to C# via `Microsoft.ML.OnnxRuntime`. End-to-end behaviour replicates the Python probe at `tests/02-wakeword/main.py`.
+
+This is a throwaway probe. Once it passes the criteria the Python probe passed, its job is done — it does not become the foundation of the main C# Pi client. The `WakewordModel.cs` class it produces is the natural lift-and-shift candidate for the eventual client, but that's a future decision, not a constraint on the probe.
+
+Test 1's record-play C# port (`tests/01-record-play-cs/`) is the immediate ancestor. Its patterns — `Libc.setenv` for env knobs the C runtime reads, callback-driven `PortAudioSharp.Stream`, `<AllowUnsafeBlocks>`, `using Stream = PortAudioSharp.Stream;` — carry over verbatim. See `findings.md` § "C# probe outcome (2026-06-12 — Test 1 ported to C#)".
+
+## Non-goals
+
+- No numerical-parity check against the Python implementation. We trust the port if hardware passes.
+- No custom wakeword training. Stock "alexa" model only. Custom wakeword is a separate task (see `findings.md` § C).
+- No fix for "speaker is the mic" self-triggering. Main-assistant concern (`findings.md` § B), out of probe scope.
+- No reusable audio abstraction. One-folder probe; if it grows beyond ~400 lines that's a smell.
+- No automated tests. Verification is the same listening test the Python probe uses.
+- No changes to `bin/bootstrap`. The Pi gets a self-contained binary; .NET is never installed on the Pi.
+- No coupling to the Python prototypes. They keep working, untouched.
+
+## Functional spec
+
+Identical to the Python Test 2 (`tests/02-wakeword/main.py:4-7`):
+
+- Open a 16 kHz mono PCM16 input stream on the USB Speaker Phone.
+- Run each 80 ms (1280-sample) frame through the openwakeword "alexa" pipeline.
+- When the classifier score crosses `THRESHOLD = 0.5` and at least `COOLDOWN_S = 1.0` seconds have passed since the last trigger, print `DETECTED alexa score={s:0.000} t={elapsed:0.1}s` and play a 200 ms 880 Hz beep through the same USB device.
+- Loop until Ctrl-C.
+
+### Pass criteria
+
+Same as the original Python spec:
+
+1. "alexa" at normal volume from ~1 m triggers within ~1 s in ≥ 8/10 attempts.
+2. ≤ 1 false positive per minute of unrelated speech (~3 minutes of reading aloud).
+3. Process stays under 50% of one core, observed via `htop` on the Pi.
+
+## Layout
+
+```
+tests/02-wakeword-cs/
+  Probe.csproj          # net9.0, RuntimeIdentifier=linux-arm64, PortAudioSharp2 + Microsoft.ML.OnnxRuntime
+  Program.cs            # entry point: env vars → init → device pick → input stream → main loop → beep dispatch
+  WakewordModel.cs      # IDisposable; loads three ONNX sessions; Predict(short[] frame1280) → float score
+  models/
+    melspectrogram.onnx     # ~14 KB, openwakeword resource
+    embedding_model.onnx    # ~10 MB, Google speech embedding model (openwakeword resource)
+    alexa.onnx              # ~1.5 MB, stock alexa classifier (openwakeword resource)
+
+bin/probe-cs-2          # build (locally) → scp (to Pi) → run (over SSH); copy of bin/probe-cs with new paths
+```
+
+The Python prototypes under `tests/01-record-play/` and `tests/02-wakeword/` and the Test 1 C# probe under `tests/01-record-play-cs/` are untouched.
+
+### Sourcing the ONNX files for the repo
+
+The three `.onnx` files are sourced from the Pi's existing openwakeword install at implementation time:
+
+```sh
+sshpass -p assistant scp \
+  pi@192.168.50.115:.local/lib/python3.13/site-packages/openwakeword/resources/models/{melspectrogram,embedding_model,alexa}.onnx \
+  tests/02-wakeword-cs/models/
+```
+
+Their SHA-256 hashes are recorded in the implementation plan for reproducibility. The same files openwakeword's Python `download_models()` helper fetches at first run — we vendor them rather than introducing a startup network dependency. ~11 MB total in git is acceptable for a probe.
+
+## Components
+
+### `WakewordModel.cs` — the substantive port
+
+Public surface:
+
+```csharp
+class WakewordModel : IDisposable {
+    WakewordModel(string melPath, string embeddingPath, string classifierPath);
+    float Predict(short[] frame1280);   // returns score in [0, 1]
+    void Dispose();
+}
+```
+
+Constructor loads three `Microsoft.ML.OnnxRuntime.InferenceSession` instances (one per ONNX file). Each session is configured with `SessionOptions { IntraOpNumThreads = 1, InterOpNumThreads = 1, LogSeverityLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR }` to keep CPU bounded and the console quiet.
+
+`Predict` is called once per 80 ms input frame (1280 PCM16 samples at 16 kHz) and returns the most recent classifier score.
+
+**Pipeline (matches `openwakeword/model.py`'s `Model.predict()`):**
+
+1. **Audio buffer.** Append the incoming 1280 samples to a raw-audio ring of recent samples.
+2. **Mel stage.** Run `melspectrogram.onnx` over the latest window of audio. Append output mel frames to a mel-frame ring.
+3. **Embedding stage.** When the mel ring has accumulated enough fresh mel frames to fill the embedding model's input window (76 frames per openwakeword's source — but verify), slice the window, run `embedding_model.onnx`. Append the resulting embedding to an embedding ring.
+4. **Classifier stage.** Slice the most recent N embeddings (N = whatever shape `alexa.onnx` declares on its input). Run the classifier. Return the score.
+
+Per 80 ms input, the classifier runs exactly once. Between calls, the three ring buffers hold all cross-call state.
+
+**Ground truth — do not guess tensor shapes.** The exact buffer sizes (raw-audio window length, mel-frame count for embedder input, embedding count for classifier input) MUST be derived during implementation from two sources:
+
+- The Python reference: `openwakeword/model.py` on the Pi at `~/.local/lib/python3.13/site-packages/openwakeword/model.py`. Read it side-by-side with the C# code. Numbers quoted in this spec are from memory and authoritative only after verification.
+- The ONNX files themselves: `InferenceSession.InputMetadata` exposes the declared input shapes. The constructor MUST print the declared input + output shapes of all three models at startup the first time the probe runs, and SHOULD assert against them to fail fast on geometry mismatch.
+
+**Threading.** `Predict` is called only from the main thread (per the data-flow section below). No internal locking. `Dispose` cleans up all three sessions.
+
+**Why three sessions instead of one combined graph.** openwakeword ships them as three separate files because the embedding model is Google's pretrained TF Hub model. Combining them offline is possible but a deliberate optimisation, not needed for the probe.
+
+### `Program.cs` — entry point and audio I/O
+
+Single top-level-statements file. Reuses every Test 1 pattern:
+
+- Set `PA_ALSA_PLUGHW=1` via `Libc.setenv("PA_ALSA_PLUGHW", "1", 1)` + `Environment.SetEnvironmentVariable` (the dual-call workaround documented in `findings.md`).
+- Set `ORT_LOGGING_LEVEL=3` (error-only) via `Libc.setenv` to suppress onnxruntime's `/sys/class/drm/card0` GPU-detection warnings on the Pi.
+- `PortAudio.Initialize()`.
+- `FindUsbDevice()` — same body as Test 1; first device whose name contains `usb` (case-insensitive) and has `maxInputChannels >= 1`. On failure: print device table to stderr, exit 1.
+- `var model = new WakewordModel(...)` — paths resolved relative to `AppContext.BaseDirectory + "models/"`. Print load time (mirrors Python's `"Model loaded in {t}s"`).
+- Pre-compute the 200 ms 880 Hz sine-wave beep as a `short[3200]` once at startup. Reused for every detection.
+- Open input stream: callback-driven, 16 kHz, 1 channel, Int16, 1280-frame block size. Callback copies into a fresh `short[1280]` and pushes into a `BlockingCollection<short[]>(boundedCapacity: 16)`. On `StreamCallbackFlags.InputOverflow`: log `[status] input overflow` to stderr. On `TryAdd` failure (queue full): log `[status] consumer behind` and drop the frame.
+- `stream.Start()`. Print `Listening on device {i} ({name}). Say 'alexa'. Ctrl-C to exit.`
+- Main loop:
+
+  ```
+  while (!cts.IsCancellationRequested) {
+      var frame = queue.Take(cts.Token);
+      float score = model.Predict(frame);
+      var now = sw.Elapsed;
+      if (score >= 0.5f && (now - lastTrigger).TotalSeconds >= 1.0) {
+          Console.WriteLine($"DETECTED alexa score={score:0.000} t={now.TotalSeconds:0.0}s");
+          lastTrigger = now;
+          FireAndForgetBeep(device, beepBuffer);
+      }
+  }
+  ```
+
+- Ctrl-C via `Console.CancelKeyPress` → `cts.Cancel()` → break out of `Take`, stop input stream, dispose `WakewordModel`, `PortAudio.Terminate()`. Graceful.
+
+### `FireAndForgetBeep(int device, short[] beepBuffer)` — the bug fix
+
+The Python probe blocks the input thread for ~200 ms on each beep (`findings.md` § A). The C# probe fixes this:
+
+- On detection, open a new output-only `Stream` instance (16 kHz, 1 channel, Int16, 1024-frame block). Callback drains `beepBuffer` at a write offset; on exhaustion sets a `ManualResetEventSlim` and returns `StreamCallbackResult.Complete`.
+- A small `Task.Run` waits the event, then `Stream.Stop()` + `Stream.Dispose()`s the stream. Main thread does not wait.
+- The 1.0 s cooldown is well past the 200 ms playback duration, so the previous beep stream has always finished disposing before the next one is opened. If for any reason it hasn't, opening a fresh one is fine — they're independent.
+
+This is the first time we run two PortAudio streams concurrently on the same USB device. Validating that this works is part of what the probe surfaces.
+
+## Data flow
+
+```
+PortAudio input callback (RT thread)
+    └─ alloc short[1280], memcpy from PortAudio buffer
+    └─ queue.TryAdd(frame, 0ms)
+        ├─ success → return Continue
+        └─ failure → log "[status] consumer behind", drop frame, return Continue
+
+Main thread
+    └─ for each frame from queue:
+        ├─ WakewordModel.Predict(frame)  # runs all three ONNX models
+        ├─ if score >= 0.5 AND cooldown ok:
+        │   ├─ print DETECTED line
+        │   └─ FireAndForgetBeep(device, beepBuffer):
+        │       ├─ create ManualResetEventSlim 'done'
+        │       ├─ create output Stream (callback drains beepBuffer, Sets 'done' on exhaustion)
+        │       ├─ Task.Run(() => { done.Wait(); stream.Stop(); stream.Dispose(); })
+        │       └─ stream.Start(); return immediately
+        └─ loop
+
+Output callback (separate RT thread, one per beep)
+    └─ drain beepBuffer → return Complete on exhaustion
+        └─ Sets 'done' event → cleanup Task wakes and disposes the stream
+```
+
+Three concerns isolated to three threads: PortAudio's input callback (allocate + enqueue only), the main thread (all inference + dispatch), per-beep PortAudio output callbacks. No locks in the steady state — the queue and the per-beep events are the synchronisation surface.
+
+## Error handling
+
+- USB device not found → device table to stderr, exit 1. (Same as Test 1.)
+- `PortAudio.Initialize`, `Stream` ctor, `Stream.Start` → `PaError != NoError` throws `PortAudioException` from PortAudioSharp; uncaught, crashes with a useful trace. (Same as Test 1.)
+- `WakewordModel` constructor:
+  - Missing ONNX file at given path → throw `FileNotFoundException` with the expected path.
+  - Loaded model's `InputMetadata` shape disagrees with what the pipeline expects → throw `InvalidOperationException` with which model and what shape was declared vs expected.
+- Input overflow flag → log `[status] input overflow` to stderr, keep going. Same as Test 1 and the Python probe.
+- Queue full at enqueue time → log `[status] consumer behind`, drop frame, keep going. Sustained occurrence is a probe finding worth documenting in `findings.md` post-run.
+- Anything else (PortAudio mid-stream device disappearance, onnxruntime inference errors, CoreCLR issues) → uncaught throw. Probe, not production.
+
+## Deploy & run (`bin/probe-cs-2`)
+
+A copy of `bin/probe-cs` with three changes: project path, remote dir name, run path. Body:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+PI_HOST="${PI_HOST:-192.168.50.115}"
+PI_USER="${PI_USER:-pi}"
+PI_PASS="${PI_PASS:-assistant}"
+PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-2-out}"
+
+repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$repo_root"
+
+echo ">> dotnet publish (linux-arm64, self-contained)"
+dotnet publish tests/02-wakeword-cs \
+  -c Release -r linux-arm64 --self-contained \
+  -o "$PUBLISH_DIR"
+
+echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-2/  (wipe-and-replace)"
+sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
+  "$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-2 && mkdir ~/probe-cs-2'
+sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
+  "$PI_USER@$PI_HOST:~/probe-cs-2/"
+
+echo ">> ssh + run on Pi"
+sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
+  "$PI_USER@$PI_HOST" 'cd ~/probe-cs-2 && chmod +x Probe && ./Probe'
+```
+
+Note `ssh -t` (pseudo-terminal) so Ctrl-C from the workstation propagates as SIGINT to the remote process for the `Console.CancelKeyPress` graceful-shutdown path. Test 1 didn't need this because the probe self-terminated.
+
+Cleanup (when the probe is retired): `ssh pi@... 'rm -rf ~/probe-cs-2'` plus `git rm -r tests/02-wakeword-cs bin/probe-cs-2`.
+
+## Verification
+
+The pass criteria above. There are no automated tests.
+
+**Hardware test protocol.**
+
+1. `bin/probe-cs-2` from the workstation. Wait for `Listening...` prompt.
+2. Position yourself ~1 m from the USB Speaker Phone.
+3. Say "alexa" at normal volume, 10 times, pausing ≥ 2 s between attempts. Count detections (printed line + beep).
+4. Read three minutes of arbitrary text aloud (newspaper or book — anything that isn't "alexa"). Count false-positive beeps.
+5. From a second terminal: `sshpass -p assistant ssh pi@192.168.50.115 htop`. Read the `Probe` process CPU% column.
+
+Pass = ≥ 8/10 detections, ≤ 1 false positive per minute, < 50% of one core. Ctrl-C to exit.
+
+### Diagnostic levers if it fails
+
+- **Detections < 8/10 but scores log near threshold.** Likely buffer geometry off-by-one in the embedding stage; re-verify `WakewordModel.cs` against `openwakeword/model.py` side-by-side.
+- **No detections, scores ≈ 0.** Tensor-shape or dtype mismatch upstream of the classifier — the startup assertion should have caught it; if it didn't, the assertion is incomplete. Fix the assertion first, then re-check.
+- **Excess false positives.** Threshold too low for this model + room. The Python probe passed at 0.5; if C# scores are systematically higher, the bug is in the port. If lower, also a port bug. Either way, do not blindly retune the threshold.
+- **CPU > 50%.** Confirm `IntraOpNumThreads = 1` was set on all sessions. Inspect with `htop` whether the spike is on input frames or on a sustained background — the latter suggests something is spinning.
+- **Input-overflow / consumer-behind warnings.** Inference latency spike. Log per-frame `WakewordModel.Predict` duration; spikes above 80 ms mean inference cannot keep up.
+
+## Open assumption (verify at implementation time)
+
+`Microsoft.ML.OnnxRuntime` NuGet package on linux-arm64 ships its own native `libonnxruntime.so` for the RID, so the self-contained publish includes it. Verify by listing the publish dir after `dotnet publish`. If for some reason it doesn't (older versions of the package split the native runtime separately), fall back to `Microsoft.ML.OnnxRuntime.Managed` + an explicit native-runtime package for the RID, or fetch the prebuilt binary from the ONNX Runtime GitHub releases. Confirmed at implementation, not now — does not affect the design.
+
+## Post-run: `findings.md` update
+
+After hardware passes, append a `## C# wakeword probe outcome (2026-06-12 — Test 2 ported to C#)` section to `findings.md` in the same shape as the Test 1 section:
+
+- What we proved (the three pass criteria, plus "two PortAudio streams on one device works", plus "ONNX runtime on Pi performant enough").
+- Key gotchas hit during the port (anything not already captured by the Test 1 section).
+- Open questions deferred to the main assistant build (custom wakeword training, AEC for speaker-is-mic, queue depth in real conditions).
+
+## References
+
+- Python probe: `tests/02-wakeword/main.py`
+- Test 1 C# probe (patterns reused): `tests/01-record-play-cs/Program.cs`, `tests/01-record-play-cs/Probe.csproj`, `bin/probe-cs`
+- openwakeword inference reference (ground truth for buffering geometry): `~/.local/lib/python3.13/site-packages/openwakeword/model.py` on the Pi
+- Prototype findings: `findings.md` — especially "C# probe outcome (2026-06-12 — Test 1 ported to C#)", "Force PortAudio through ALSA's `plug` plugin", "Beep / TTS blocks the input stream", "Cosmetic / known-warnings" (ORT GPU detection)
+- Microsoft.ML.OnnxRuntime — fetch current package docs via Context7 at implementation time (`/microsoft/onnxruntime`).
+- Test 1 C# probe spec / plan (precedent for this spec's shape): `docs/superpowers/specs/2026-06-12-record-play-csharp-probe-design.md`, `docs/superpowers/plans/2026-06-12-record-play-csharp-probe.md`
+- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)