Assistant/docs/superpowers/specs/2026-06-12-test-2-wakeword-csharp-design.md

Test 2 wakeword C# probe — design

Date: 2026-06-12 Branch: fresh-start Status: approved (pending written-spec review)

Purpose

Prove that openwakeword's stock "alexa" detector can be driven from .NET on the Pi, by porting openwakeword/model.py's predict() pipeline (three ONNX models chained: mel-spectrogram → speech embedding → keyword classifier) to C# via Microsoft.ML.OnnxRuntime. End-to-end behaviour replicates the Python probe at tests/02-wakeword/main.py.

This is a throwaway probe. Once it passes the criteria the Python probe passed, its job is done — it does not become the foundation of the main C# Pi client. The WakewordModel.cs class it produces is the natural lift-and-shift candidate for the eventual client, but that's a future decision, not a constraint on the probe.

Test 1's record-play C# port (tests/01-record-play-cs/) is the immediate ancestor. Its patterns — Libc.setenv for env knobs the C runtime reads, callback-driven PortAudioSharp.Stream, <AllowUnsafeBlocks>, using Stream = PortAudioSharp.Stream; — carry over verbatim. See findings.md § "C# probe outcome (2026-06-12 — Test 1 ported to C#)".

Non-goals

• No numerical-parity check against the Python implementation. We trust the port if hardware passes.
• No custom wakeword training. Stock "alexa" model only. Custom wakeword is a separate task (see findings.md § C).
• No fix for "speaker is the mic" self-triggering. Main-assistant concern (findings.md § B), out of probe scope.
• No reusable audio abstraction. One-folder probe; if it grows beyond ~400 lines that's a smell.
• No automated tests. Verification is the same listening test the Python probe uses.
• No changes to bin/bootstrap. The Pi gets a self-contained binary; .NET is never installed on the Pi.
• No coupling to the Python prototypes. They keep working, untouched.

Functional spec

Identical to the Python Test 2 (tests/02-wakeword/main.py:4-7):

• Open a 16 kHz mono PCM16 input stream on the USB Speaker Phone.
• Run each 80 ms (1280-sample) frame through the openwakeword "alexa" pipeline.
• When the classifier score crosses THRESHOLD = 0.5 and at least COOLDOWN_S = 1.0 seconds have passed since the last trigger, print DETECTED alexa score={s:0.000} t={elapsed:0.1}s and play a 200 ms 880 Hz beep through the same USB device.
• Loop until Ctrl-C.

Pass criteria

Same as the original Python spec:

1. "alexa" at normal volume from ~1 m triggers within ~1 s in ≥ 8/10 attempts.
2. ≤ 1 false positive per minute of unrelated speech (~3 minutes of reading aloud).
3. Process stays under 50% of one core, observed via htop on the Pi.

Layout

tests/02-wakeword-cs/
  Probe.csproj          # net9.0, RuntimeIdentifier=linux-arm64, PortAudioSharp2 + Microsoft.ML.OnnxRuntime
  Program.cs            # entry point: env vars → init → device pick → input stream → main loop → beep dispatch
  WakewordModel.cs      # IDisposable; loads three ONNX sessions; Predict(short[] frame1280) → float score
  models/
    melspectrogram.onnx     # ~14 KB, openwakeword resource
    embedding_model.onnx    # ~10 MB, Google speech embedding model (openwakeword resource)
    alexa.onnx              # ~1.5 MB, stock alexa classifier (openwakeword resource)

bin/probe-cs-2          # build (locally) → scp (to Pi) → run (over SSH); copy of bin/probe-cs with new paths

The Python prototypes under tests/01-record-play/ and tests/02-wakeword/ and the Test 1 C# probe under tests/01-record-play-cs/ are untouched.

Sourcing the ONNX files for the repo

The three .onnx files are sourced from the Pi's existing openwakeword install at implementation time:

sshpass -p assistant scp \
  pi@192.168.50.115:.local/lib/python3.13/site-packages/openwakeword/resources/models/{melspectrogram,embedding_model,alexa}.onnx \
  tests/02-wakeword-cs/models/

Their SHA-256 hashes are recorded in the implementation plan for reproducibility. The same files openwakeword's Python download_models() helper fetches at first run — we vendor them rather than introducing a startup network dependency. ~11 MB total in git is acceptable for a probe.

Components

WakewordModel.cs — the substantive port

Public surface:

class WakewordModel : IDisposable {
    WakewordModel(string melPath, string embeddingPath, string classifierPath);
    float Predict(short[] frame1280);   // returns score in [0, 1]
    void Dispose();
}

Constructor loads three Microsoft.ML.OnnxRuntime.InferenceSession instances (one per ONNX file). Each session is configured with SessionOptions { IntraOpNumThreads = 1, InterOpNumThreads = 1, LogSeverityLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR } to keep CPU bounded and the console quiet.

Predict is called once per 80 ms input frame (1280 PCM16 samples at 16 kHz) and returns the most recent classifier score.

Pipeline (matches openwakeword/model.py's Model.predict()):

1. Audio buffer. Append the incoming 1280 samples to a raw-audio ring of recent samples.
2. Mel stage. Run melspectrogram.onnx over the latest window of audio. Append output mel frames to a mel-frame ring.
3. Embedding stage. When the mel ring has accumulated enough fresh mel frames to fill the embedding model's input window (76 frames per openwakeword's source — but verify), slice the window, run embedding_model.onnx. Append the resulting embedding to an embedding ring.
4. Classifier stage. Slice the most recent N embeddings (N = whatever shape alexa.onnx declares on its input). Run the classifier. Return the score.

Per 80 ms input, the classifier runs exactly once. Between calls, the three ring buffers hold all cross-call state.

Ground truth — do not guess tensor shapes. The exact buffer sizes (raw-audio window length, mel-frame count for embedder input, embedding count for classifier input) MUST be derived during implementation from two sources:

• The Python reference: openwakeword/model.py on the Pi at ~/.local/lib/python3.13/site-packages/openwakeword/model.py. Read it side-by-side with the C# code. Numbers quoted in this spec are from memory and authoritative only after verification.
• The ONNX files themselves: InferenceSession.InputMetadata exposes the declared input shapes. The constructor MUST print the declared input + output shapes of all three models at startup the first time the probe runs, and SHOULD assert against them to fail fast on geometry mismatch.

Threading. Predict is called only from the main thread (per the data-flow section below). No internal locking. Dispose cleans up all three sessions.

Why three sessions instead of one combined graph. openwakeword ships them as three separate files because the embedding model is Google's pretrained TF Hub model. Combining them offline is possible but a deliberate optimisation, not needed for the probe.

Program.cs — entry point and audio I/O

Single top-level-statements file. Reuses every Test 1 pattern:

• Set PA_ALSA_PLUGHW=1 via Libc.setenv("PA_ALSA_PLUGHW", "1", 1) + Environment.SetEnvironmentVariable (the dual-call workaround documented in findings.md).

• Set ORT_LOGGING_LEVEL=3 (error-only) via Libc.setenv to suppress onnxruntime's /sys/class/drm/card0 GPU-detection warnings on the Pi.

• PortAudio.Initialize().

• FindUsbDevice() — same body as Test 1; first device whose name contains usb (case-insensitive) and has maxInputChannels >= 1. On failure: print device table to stderr, exit 1.

• var model = new WakewordModel(...) — paths resolved relative to AppContext.BaseDirectory + "models/". Print load time (mirrors Python's "Model loaded in {t}s").

• Pre-compute the 200 ms 880 Hz sine-wave beep as a short[3200] once at startup. Reused for every detection.

• Open input stream: callback-driven, 16 kHz, 1 channel, Int16, 1280-frame block size. Callback copies into a fresh short[1280] and pushes into a BlockingCollection<short[]>(boundedCapacity: 16). On StreamCallbackFlags.InputOverflow: log [status] input overflow to stderr. On TryAdd failure (queue full): log [status] consumer behind and drop the frame.

• stream.Start(). Print Listening on device {i} ({name}). Say 'alexa'. Ctrl-C to exit.

• Main loop:

  while (!cts.IsCancellationRequested) {
      var frame = queue.Take(cts.Token);
      float score = model.Predict(frame);
      var now = sw.Elapsed;
      if (score >= 0.5f && (now - lastTrigger).TotalSeconds >= 1.0) {
          Console.WriteLine($"DETECTED alexa score={score:0.000} t={now.TotalSeconds:0.0}s");
          lastTrigger = now;
          FireAndForgetBeep(device, beepBuffer);
      }
  }
  

• Ctrl-C via Console.CancelKeyPress → cts.Cancel() → break out of Take, stop input stream, dispose WakewordModel, PortAudio.Terminate(). Graceful.

FireAndForgetBeep(int device, short[] beepBuffer) — the bug fix

The Python probe blocks the input thread for ~200 ms on each beep (findings.md § A). The C# probe fixes this:

• On detection, open a new output-only Stream instance (16 kHz, 1 channel, Int16, 1024-frame block). Callback drains beepBuffer at a write offset; on exhaustion sets a ManualResetEventSlim and returns StreamCallbackResult.Complete.
• A small Task.Run waits the event, then Stream.Stop() + Stream.Dispose()s the stream. Main thread does not wait.
• The 1.0 s cooldown is well past the 200 ms playback duration, so the previous beep stream has always finished disposing before the next one is opened. If for any reason it hasn't, opening a fresh one is fine — they're independent.

This is the first time we run two PortAudio streams concurrently on the same USB device. Validating that this works is part of what the probe surfaces.

Data flow

PortAudio input callback (RT thread)
    └─ alloc short[1280], memcpy from PortAudio buffer
    └─ queue.TryAdd(frame, 0ms)
        ├─ success → return Continue
        └─ failure → log "[status] consumer behind", drop frame, return Continue

Main thread
    └─ for each frame from queue:
        ├─ WakewordModel.Predict(frame)  # runs all three ONNX models
        ├─ if score >= 0.5 AND cooldown ok:
        │   ├─ print DETECTED line
        │   └─ FireAndForgetBeep(device, beepBuffer):
        │       ├─ create ManualResetEventSlim 'done'
        │       ├─ create output Stream (callback drains beepBuffer, Sets 'done' on exhaustion)
        │       ├─ Task.Run(() => { done.Wait(); stream.Stop(); stream.Dispose(); })
        │       └─ stream.Start(); return immediately
        └─ loop

Output callback (separate RT thread, one per beep)
    └─ drain beepBuffer → return Complete on exhaustion
        └─ Sets 'done' event → cleanup Task wakes and disposes the stream

Three concerns isolated to three threads: PortAudio's input callback (allocate + enqueue only), the main thread (all inference + dispatch), per-beep PortAudio output callbacks. No locks in the steady state — the queue and the per-beep events are the synchronisation surface.

Error handling

• USB device not found → device table to stderr, exit 1. (Same as Test 1.)
• PortAudio.Initialize, Stream ctor, Stream.Start → PaError != NoError throws PortAudioException from PortAudioSharp; uncaught, crashes with a useful trace. (Same as Test 1.)
• WakewordModel constructor:
  • Missing ONNX file at given path → throw FileNotFoundException with the expected path.
  • Loaded model's InputMetadata shape disagrees with what the pipeline expects → throw InvalidOperationException with which model and what shape was declared vs expected.
• Input overflow flag → log [status] input overflow to stderr, keep going. Same as Test 1 and the Python probe.
• Queue full at enqueue time → log [status] consumer behind, drop frame, keep going. Sustained occurrence is a probe finding worth documenting in findings.md post-run.
• Anything else (PortAudio mid-stream device disappearance, onnxruntime inference errors, CoreCLR issues) → uncaught throw. Probe, not production.

Deploy & run (bin/probe-cs-2)

A copy of bin/probe-cs with three changes: project path, remote dir name, run path. Body:

#!/usr/bin/env bash
set -euo pipefail

PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-2-out}"

repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"

echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/02-wakeword-cs \
  -c Release -r linux-arm64 --self-contained \
  -o "$PUBLISH_DIR"

echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-2/  (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
  "$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-2 && mkdir ~/probe-cs-2'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
  "$PI_USER@$PI_HOST:~/probe-cs-2/"

echo ">> ssh + run on Pi"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
  "$PI_USER@$PI_HOST" 'cd ~/probe-cs-2 && chmod +x Probe && ./Probe'

Note ssh -t (pseudo-terminal) so Ctrl-C from the workstation propagates as SIGINT to the remote process for the Console.CancelKeyPress graceful-shutdown path. Test 1 didn't need this because the probe self-terminated.

Cleanup (when the probe is retired): ssh pi@... 'rm -rf ~/probe-cs-2' plus git rm -r tests/02-wakeword-cs bin/probe-cs-2.

Verification

The pass criteria above. There are no automated tests.

Hardware test protocol.

1. bin/probe-cs-2 from the workstation. Wait for Listening... prompt.
2. Position yourself ~1 m from the USB Speaker Phone.
3. Say "alexa" at normal volume, 10 times, pausing ≥ 2 s between attempts. Count detections (printed line + beep).
4. Read three minutes of arbitrary text aloud (newspaper or book — anything that isn't "alexa"). Count false-positive beeps.
5. From a second terminal: sshpass -p assistant ssh pi@192.168.50.115 htop. Read the Probe process CPU% column.

Pass = ≥ 8/10 detections, ≤ 1 false positive per minute, < 50% of one core. Ctrl-C to exit.

Diagnostic levers if it fails

• Detections < 8/10 but scores log near threshold. Likely buffer geometry off-by-one in the embedding stage; re-verify WakewordModel.cs against openwakeword/model.py side-by-side.
• No detections, scores ≈ 0. Tensor-shape or dtype mismatch upstream of the classifier — the startup assertion should have caught it; if it didn't, the assertion is incomplete. Fix the assertion first, then re-check.
• Excess false positives. Threshold too low for this model + room. The Python probe passed at 0.5; if C# scores are systematically higher, the bug is in the port. If lower, also a port bug. Either way, do not blindly retune the threshold.
• CPU > 50%. Confirm IntraOpNumThreads = 1 was set on all sessions. Inspect with htop whether the spike is on input frames or on a sustained background — the latter suggests something is spinning.
• Input-overflow / consumer-behind warnings. Inference latency spike. Log per-frame WakewordModel.Predict duration; spikes above 80 ms mean inference cannot keep up.

Open assumption (verify at implementation time)

Microsoft.ML.OnnxRuntime NuGet package on linux-arm64 ships its own native libonnxruntime.so for the RID, so the self-contained publish includes it. Verify by listing the publish dir after dotnet publish. If for some reason it doesn't (older versions of the package split the native runtime separately), fall back to Microsoft.ML.OnnxRuntime.Managed + an explicit native-runtime package for the RID, or fetch the prebuilt binary from the ONNX Runtime GitHub releases. Confirmed at implementation, not now — does not affect the design.

Post-run: findings.md update

After hardware passes, append a ## C# wakeword probe outcome (2026-06-12 — Test 2 ported to C#) section to findings.md in the same shape as the Test 1 section:

• What we proved (the three pass criteria, plus "two PortAudio streams on one device works", plus "ONNX runtime on Pi performant enough").
• Key gotchas hit during the port (anything not already captured by the Test 1 section).
• Open questions deferred to the main assistant build (custom wakeword training, AEC for speaker-is-mic, queue depth in real conditions).

References

• Python probe: tests/02-wakeword/main.py
• Test 1 C# probe (patterns reused): tests/01-record-play-cs/Program.cs, tests/01-record-play-cs/Probe.csproj, bin/probe-cs
• openwakeword inference reference (ground truth for buffering geometry): ~/.local/lib/python3.13/site-packages/openwakeword/model.py on the Pi
• Prototype findings: findings.md — especially "C# probe outcome (2026-06-12 — Test 1 ported to C#)", "Force PortAudio through ALSA's plug plugin", "Beep / TTS blocks the input stream", "Cosmetic / known-warnings" (ORT GPU detection)
• Microsoft.ML.OnnxRuntime — fetch current package docs via Context7 at implementation time (/microsoft/onnxruntime).
• Test 1 C# probe spec / plan (precedent for this spec's shape): docs/superpowers/specs/2026-06-12-record-play-csharp-probe-design.md, docs/superpowers/plans/2026-06-12-record-play-csharp-probe.md
• Pi access: pi@192.168.50.115, password assistant (see CLAUDE.md)