Compare commits

...

10 Commits

Author SHA1 Message Date
tes cbb47d1465 Findings: Test 3 C# full-cycle probe outcome 2026-06-12 13:56:16 +00:00
tes a7d408e903 Test 3: scale playback timeout to buffer length (was hardcoded 2 s)
FireAndForgetPlayback's cleanup task had the same 2 s done.Wait timeout
as the beep cleanup. Beep is 200 ms so 2 s is generous; recording
playback is 5 s so the timeout fired mid-playback, cleanup ran
stream.Stop(), and the audio cut at ~2.2 s. Timeout is now
recording-duration + 2 s.
2026-06-12 13:54:16 +00:00
tes 1d9a417cfc Test 3: bin/probe-cs-3 deploy script 2026-06-12 13:40:55 +00:00
tes 070cc0767a Test 3: per-cycle PlaybackDoneFlag + log timeout in fire-and-forget cleanup
Stale Set() from a delayed cleanup task could short-circuit a future
PLAYBACK state; new flag per cycle avoids the cross-cycle leak. Also
log when the 2 s done.Wait timeout fires so silent PortAudio callback
faults are visible during hardware debugging.
2026-06-12 13:38:09 +00:00
tes 5a498e9c91 Test 3: full-cycle state machine (IDLE/BEEPING/RECORDING/PLAYBACK) 2026-06-12 13:31:25 +00:00
tes 222d58baeb Test 3: copy WakewordModel + models from Test 2, add Reset() 2026-06-12 13:25:40 +00:00
tes 6fa2779ea1 Test 3 scaffold: csproj for tests/03-full-cycle-cs 2026-06-12 13:22:23 +00:00
tes 14690484de Test 3 plan: implementation steps for full-cycle C# probe
Five tasks: scaffold csproj, copy WakewordModel + models with Reset(),
write Program.cs with the four-state machine, write bin/probe-cs-3
deploy script, and hardware verification on the Pi.
2026-06-12 13:19:18 +00:00
tes e9730c7e50 Test 3 spec: full-cycle C# probe (wakeword + record + playback)
State machine on a single persistent input stream: IDLE → BEEPING →
RECORDING → PLAYBACK → IDLE. Detector gated during non-IDLE states;
WakewordModel.Reset() called on IDLE re-entry to clear stale ring
buffers and re-arm the warmup guard.
2026-06-12 13:13:59 +00:00
tes b6d9f9a1cf WakewordModel: print shapes at startup + tidy mel-shape comment
Add PrintShapes() helper that writes input/output name, shape, and dtype
for each InferenceSession to stderr at construction time, prefixed with
[wakeword-model]. Called after each of the three session creations so the
geometry is visible on first run without grep-ing logs.

Also rephrase the stale "FIXED:" comment on the mel Dimensions[2] indexing
to plain explanatory text that describes the current invariant.
2026-06-12 12:19:59 +00:00
11 changed files with 1722 additions and 1 deletions
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-3-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/03-full-cycle-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-3/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-3 && mkdir ~/probe-cs-3'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-3/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-3 && chmod +x Probe && ./Probe'
@@ -0,0 +1,758 @@
# Test 3 — Full-cycle C# probe Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a single-binary C# probe at `tests/03-full-cycle-cs/` that runs the full assistant cycle on the Pi: idle → "alexa" → beep → record 5 s → play back → idle, with detector gating during non-idle states and a `WakewordModel.Reset()` on every IDLE re-entry.
**Architecture:** One persistent PortAudio input stream feeds 80 ms frames into a `BlockingCollection`; a single consumer thread runs a four-state machine (IDLE / BEEPING / RECORDING / PLAYBACK). Beep and playback are fire-and-forget output streams using Test 2's `Start()`-before-`Task.Run` + 2 s timeout lifetime pattern. `WakewordModel` is copied verbatim from Test 2 with a new `Reset()` method called on PLAYBACK→IDLE.
**Tech Stack:** .NET 9 (self-contained linux-arm64), PortAudioSharp2 1.0.6, Microsoft.ML.OnnxRuntime 1.26.0. Deploy via `dotnet publish` + `scp` + `ssh -t` to Pi at `pi@192.168.50.115`.
**Spec:** `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`. Read it before starting.
**Branch:** `fresh-start` (already current; do NOT create a new branch).
---
### Task 1: Scaffold `tests/03-full-cycle-cs/` project skeleton
**Files:**
- Create: `tests/03-full-cycle-cs/Probe.csproj`
- Create: `tests/03-full-cycle-cs/models/` (directory; populated in Task 2)
- [ ] **Step 1: Create the project directory**
```bash
mkdir -p tests/03-full-cycle-cs/models
```
- [ ] **Step 2: Write `Probe.csproj`**
Create `tests/03-full-cycle-cs/Probe.csproj` with this exact content (identical to `tests/02-wakeword-cs/Probe.csproj` except `RootNamespace`):
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>FullCycleProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.26.0" />
</ItemGroup>
<ItemGroup>
<Content Include="models/*.onnx">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project>
```
- [ ] **Step 3: Commit the scaffold**
```bash
git add tests/03-full-cycle-cs/Probe.csproj
git commit -m "Test 3 scaffold: csproj for tests/03-full-cycle-cs"
```
---
### Task 2: Copy models and `WakewordModel.cs` from Test 2 (with namespace rename and `Reset()`)
**Files:**
- Create: `tests/03-full-cycle-cs/WakewordModel.cs` (copied from Test 2, namespace renamed, `Reset()` added)
- Create: `tests/03-full-cycle-cs/models/melspectrogram.onnx`
- Create: `tests/03-full-cycle-cs/models/embedding_model.onnx`
- Create: `tests/03-full-cycle-cs/models/alexa.onnx`
- [ ] **Step 1: Copy the three ONNX model files**
```bash
cp tests/02-wakeword-cs/models/melspectrogram.onnx tests/03-full-cycle-cs/models/
cp tests/02-wakeword-cs/models/embedding_model.onnx tests/03-full-cycle-cs/models/
cp tests/02-wakeword-cs/models/alexa.onnx tests/03-full-cycle-cs/models/
```
- [ ] **Step 2: Copy `WakewordModel.cs`**
```bash
cp tests/02-wakeword-cs/WakewordModel.cs tests/03-full-cycle-cs/WakewordModel.cs
```
- [ ] **Step 3: Rename the namespace in the copy**
Edit `tests/03-full-cycle-cs/WakewordModel.cs`: change line 4 from `namespace WakewordProbe;` to `namespace FullCycleProbe;`. Leave everything else unchanged.
- [ ] **Step 4: Add `Reset()` method to the copy**
Edit `tests/03-full-cycle-cs/WakewordModel.cs`. Insert the following method immediately before the `public void Dispose()` method near the bottom of the file:
```csharp
public void Reset()
{
_rawRingFill = 0;
_melRing.Clear();
_embRing.Clear();
_framesSeen = 0;
}
```
(Note the trailing blank line. The result should be a method block sitting between `return clsOut[0, 0]; }` (end of `Predict`) and `public void Dispose()`.)
- [ ] **Step 5: Verify the project builds**
Run from repo root:
```bash
dotnet build tests/03-full-cycle-cs -c Release
```
Expected: `Build succeeded` with 0 errors, 0 warnings. (At this point `Program.cs` doesn't exist yet; csproj is `<OutputType>Exe</OutputType>` but the build will produce a no-entry-point warning or error. If it errors with "CS5001: Program does not contain a static 'Main' method", that's expected at this stage — proceed to Task 3 which adds `Program.cs`.)
If the build errors for any reason other than the missing entry point, fix before continuing.
- [ ] **Step 6: Commit**
```bash
git add tests/03-full-cycle-cs/WakewordModel.cs tests/03-full-cycle-cs/models/
git commit -m "Test 3: copy WakewordModel + models from Test 2, add Reset()"
```
---
### Task 3: Write `Program.cs` with the full state machine
**Files:**
- Create: `tests/03-full-cycle-cs/Program.cs`
The whole file is shown below. It lifts from `tests/02-wakeword-cs/Program.cs` (env-var setup, OrtEnv init, FindUsbDevice, input stream + queue, FireAndForgetBeep, Libc, MakeBeep) and adds:
- `volatile State _state` enum-as-state-machine in the consumer loop.
- `FireAndForgetPlayback` helper (same shape as `FireAndForgetBeep` but reads from `recordBuf` and signals via `_playbackDoneFlag` instead of a `ManualResetEventSlim`).
- `model.Reset()` call on PLAYBACK→IDLE transition.
- [ ] **Step 1: Write `Program.cs` in full**
Create `tests/03-full-cycle-cs/Program.cs` with exactly this content:
```csharp
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Runtime.InteropServices;
using FullCycleProbe;
using Microsoft.ML.OnnxRuntime;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
const int SampleRate = WakewordModel.SampleRate; // 16 000
const int Channels = 1;
const uint BlockFrames = WakewordModel.FrameSamples; // 1280 (80 ms)
const float Threshold = 0.5f;
const double CooldownSeconds = 1.0;
const double BeepHz = 880.0;
const double BeepSeconds = 0.2;
const int BeepingFrames = 3; // 3 * 80 ms = 240 ms (covers 200 ms beep + 40 ms drain/reverb margin)
const int RecordingFrames = 63; // 63 * 80 ms = 5.04 s (rounded up from 5 s to avoid partial-frame handling)
const int RecordBufSamples = RecordingFrames * (int)BlockFrames; // 80640
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio / onnxruntime (which use getenv()) see the values.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
Environment.SetEnvironmentVariable("ORT_LOGGING_LEVEL", "3");
Libc.setenv("ORT_LOGGING_LEVEL", "3", 1);
// ORT 1.26.0: the env var does NOT suppress early GPU-discovery warnings from device_discovery.cc.
// CreateInstanceWithOptions sets the log level at OrtEnv creation time, before EP discovery runs.
var ortEnvOpts = new EnvironmentCreationOptions { logId = "FullCycleProbe", logLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR };
OrtEnv.CreateInstanceWithOptions(ref ortEnvOpts);
PortAudio.Initialize();
WakewordModel? model = null;
try
{
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
Console.WriteLine("Loading WakewordModel...");
var t0 = DateTime.UtcNow;
string modelsDir = Path.Combine(AppContext.BaseDirectory, "models");
model = new WakewordModel(
Path.Combine(modelsDir, "melspectrogram.onnx"),
Path.Combine(modelsDir, "embedding_model.onnx"),
Path.Combine(modelsDir, "alexa.onnx"));
Console.WriteLine($"Loaded in {(DateTime.UtcNow - t0).TotalSeconds:0.00}s.");
var queue = new BlockingCollection<short[]>(boundedCapacity: 16);
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var inParams = new StreamParameters
{
device = device,
channelCount = Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (frameCount != BlockFrames)
{
Console.Error.WriteLine($"[status] unexpected callback frameCount={frameCount} (want {BlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[BlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
if (!queue.TryAdd(frame, 0))
Console.Error.WriteLine("[status] consumer behind, dropping frame");
return StreamCallbackResult.Continue;
};
using var stream = new Stream(
inParams, null, SampleRate, BlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
stream.Start();
short[] beepBuffer = MakeBeep(BeepHz, BeepSeconds, SampleRate);
Console.WriteLine("Listening. Say 'alexa'. Ctrl-C to exit.");
var sw = Stopwatch.StartNew();
TimeSpan lastTrigger = TimeSpan.FromSeconds(-CooldownSeconds);
// State machine (single-writer: this consumer thread).
State state = State.Idle;
int beepFrames = 0;
int recordOffset = 0;
short[] recordBuf = null!; // assigned on BEEPING→RECORDING transition
var playbackDone = new PlaybackDoneFlag();
try
{
while (!cts.IsCancellationRequested)
{
short[] frame = queue.Take(cts.Token);
switch (state)
{
case State.Idle:
{
float score = model.Predict(frame);
var now = sw.Elapsed;
if (score >= Threshold && (now - lastTrigger).TotalSeconds >= CooldownSeconds)
{
Console.WriteLine($"DETECTED alexa score={score:0.000} t={now.TotalSeconds:0.0}s");
lastTrigger = now;
try
{
FireAndForgetBeep(device, beepBuffer);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] beep stream failed to start: {ex.Message}");
// Stay in IDLE — no cycle to abort yet.
break;
}
Console.WriteLine("[state] BEEPING");
state = State.Beeping;
beepFrames = 0;
}
break;
}
case State.Beeping:
{
beepFrames++;
if (beepFrames >= BeepingFrames)
{
Console.WriteLine("[state] RECORDING");
state = State.Recording;
recordBuf = new short[RecordBufSamples];
recordOffset = 0;
}
break;
}
case State.Recording:
{
unsafe
{
fixed (short* src = &frame[0])
fixed (short* dst = &recordBuf[recordOffset])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
recordOffset += (int)BlockFrames;
if (recordOffset >= RecordBufSamples)
{
try
{
FireAndForgetPlayback(device, recordBuf, playbackDone);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] playback stream failed to start: {ex.Message}");
// Drop the recording, reset model, return to IDLE.
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
break;
}
Console.WriteLine("[state] PLAYBACK");
state = State.Playback;
}
break;
}
case State.Playback:
{
// Discard input during playback (gate the detector).
if (playbackDone.Consume())
{
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
}
break;
}
}
}
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
stream.Stop();
Console.WriteLine("\nbye");
}
finally
{
model?.Dispose();
PortAudio.Terminate();
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static void FireAndForgetBeep(int device, short[] beepBuffer)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = beepBuffer.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &beepBuffer[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= beepBuffer.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
Task.Run(() =>
{
// 2 s timeout so a failed callback (which PortAudio silently swallows
// and would leave 'done' unset) eventually releases the stream + handle
// instead of leaking them for the process lifetime.
done.Wait(TimeSpan.FromSeconds(2));
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
});
}
static void FireAndForgetPlayback(int device, short[] recordBuf, PlaybackDoneFlag doneFlag)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = recordBuf.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &recordBuf[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= recordBuf.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
Task.Run(() =>
{
// Same 2 s safety timeout as the beep cleanup.
done.Wait(TimeSpan.FromSeconds(2));
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
doneFlag.Set(); // tell the consumer it's safe to flip PLAYBACK→IDLE
});
}
enum State { Idle, Beeping, Recording, Playback }
// Volatile flag set by the playback cleanup task and consumed by the consumer thread.
// Wrapped in a tiny class so we can pass it by reference into the helper.
sealed class PlaybackDoneFlag
{
private volatile bool _set;
public void Set() => _set = true;
public bool Consume()
{
if (!_set) return false;
_set = false;
return true;
}
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}
```
- [ ] **Step 2: Verify the project builds cleanly**
```bash
dotnet build tests/03-full-cycle-cs -c Release
```
Expected: `Build succeeded` with 0 errors, 0 warnings.
If `<AllowUnsafeBlocks>` is missing from the csproj, the build will fail with `CS0227: Unsafe code may only appear if compiling with /unsafe`. Confirm Task 1's csproj has `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`.
If you see `CS0246: The type or namespace 'Stream' could not be found` or a clash with `System.IO.Stream`, confirm the `using Stream = PortAudioSharp.Stream;` line is at the top of `Program.cs`.
- [ ] **Step 3: Verify `dotnet publish` produces a runnable binary**
```bash
dotnet publish tests/03-full-cycle-cs -c Release -r linux-arm64 --self-contained -o /tmp/probe-cs-3-out
ls /tmp/probe-cs-3-out/Probe
ls /tmp/probe-cs-3-out/models/
```
Expected:
- `Probe` (executable) exists.
- `models/` directory contains `melspectrogram.onnx`, `embedding_model.onnx`, `alexa.onnx`.
- [ ] **Step 4: Commit**
```bash
git add tests/03-full-cycle-cs/Program.cs
git commit -m "Test 3: full-cycle state machine (IDLE/BEEPING/RECORDING/PLAYBACK)"
```
---
### Task 4: Write `bin/probe-cs-3` deploy script
**Files:**
- Create: `bin/probe-cs-3`
- [ ] **Step 1: Write the deploy script**
Create `bin/probe-cs-3` with exactly this content (it's `bin/probe-cs-2` with three string swaps: `02-wakeword-cs``03-full-cycle-cs`, `probe-cs-2``probe-cs-3`, `/tmp/probe-cs-2-out``/tmp/probe-cs-3-out`):
```bash
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-3-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/03-full-cycle-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-3/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-3 && mkdir ~/probe-cs-3'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-3/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-3 && chmod +x Probe && ./Probe'
```
- [ ] **Step 2: Make it executable**
```bash
chmod +x bin/probe-cs-3
```
- [ ] **Step 3: Commit**
```bash
git add bin/probe-cs-3
git commit -m "Test 3: bin/probe-cs-3 deploy script"
```
---
### Task 5: Hardware verification on the Pi
This task is the actual pass/fail gate. No automated tests; verification is the human ear + console output + `top` reading on the Pi. If anything in this task fails, do NOT mark the implementation complete — investigate and either fix in code or update the spec to reflect what actually works.
**Files:** (none — this is a runtime verification task)
- [ ] **Step 1: Confirm the Pi is reachable**
Run from workstation:
```bash
ping -c 2 192.168.50.115
```
Expected: 2/2 replies. If unreachable, stop and ask the user.
- [ ] **Step 2: Deploy + run**
Run from workstation:
```bash
./bin/probe-cs-3
```
Expected console output (abridged):
```
>> dotnet publish (linux-arm64, self-contained)
... (build output) ...
>> scp to pi@192.168.50.115:~/probe-cs-3/ (wipe-and-replace)
>> ssh + run on Pi (Ctrl-C from this terminal stops the probe)
[wakeword-model] melspectrogram.onnx
input ...
[wakeword-model] embedding_model.onnx ...
[wakeword-model] alexa.onnx ...
Loaded in 0.6s.
Using device <N> ('USB Speaker Phone...')
Listening. Say 'alexa'. Ctrl-C to exit.
```
If the publish step errors with `setenv: command not found` or similar libc-related runtime crash, confirm Task 3's `Libc.setenv` `[DllImport]` is present and the binary was self-contained-published.
If the Pi run errors with `Cannot open shared library libonnxruntime.so` or `libportaudio.so`, the self-contained publish didn't bundle the native libs — confirm `<SelfContained>true</SelfContained>` and `<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>` in Task 1's csproj.
- [ ] **Step 3: Run 3 full cycles back-to-back**
Say "alexa" clearly at normal volume from ~1 m. Observe:
1. Beep plays within ~100 ms of saying "alexa".
2. Console prints `DETECTED alexa score=…`, then `[state] BEEPING`, then `[state] RECORDING`.
3. Say something distinctive (e.g. "hello world, this is cycle one") within the 5 s recording window.
4. Console prints `[state] PLAYBACK`.
5. Speaker plays back what you said. It should be recognisable.
6. Console prints `[state] IDLE`.
7. Pause ~1.5 s, then say "alexa" again. Verify the cycle starts over.
Repeat for cycle 2 and cycle 3. Record on paper (or transcript) which cycles succeeded.
Pass criterion 1: 3 full cycles back-to-back; each playback recognisable as what was spoken.
- [ ] **Step 4: Measure idle CPU on the Pi**
In a second workstation terminal:
```bash
sshpass -p 'assistant' ssh pi@192.168.50.115 'top -b -n 3 -d 1 -p $(pgrep -f Probe)'
```
Expected: `Probe` process appears with `%CPU` < 50 across the three samples (Test 2 measured 1731%, Test 3 should be in the same range during IDLE).
Pass criterion 4: CPU < 50% of one core during IDLE.
- [ ] **Step 5: Test post-playback re-detection latency**
Speak "alexa" immediately after each `[state] IDLE` log line. Note how soon detection fires.
Pass criterion 2: Next "alexa" after playback detected within ~2 s of playback ending. (Below ~1.3 s is impossible due to WarmupFrames; above ~2 s indicates a regression.)
- [ ] **Step 6: Scan for warnings**
Re-read the console output from Step 3. Confirm there are **no** lines of the form:
- `[status] input overflow`
- `[status] consumer behind, dropping frame`
- `[status] unexpected callback frameCount=...`
- `[error] beep stream failed to start: ...`
- `[error] playback stream failed to start: ...`
A single occurrence under abnormal load may be tolerable; sustained occurrences during normal cycles fail criterion 5.
- [ ] **Step 7: Test Ctrl-C clean exit**
Press Ctrl-C in the workstation terminal that's running the probe.
Expected:
```
^C
bye
$
```
Pass criterion 7: process exits 0, no stack trace, terminal returns to prompt. Verify with `echo $?``0`.
- [ ] **Step 8: Wakeword detection rate during IDLE**
Restart the probe (`./bin/probe-cs-3`). Wait until `Listening. Say 'alexa'. Ctrl-C to exit.` appears. Say "alexa" 10 times with a ~2 s gap between each. Count how many times `DETECTED alexa score=…` appears.
Pass criterion 3: ≥ 8/10. (Test 2 demonstrated this rate; Test 3 should match.)
- [ ] **Step 9: Update `findings.md` with the outcome**
Add a new section `## Test 3 full-cycle outcome (2026-06-12)` at the bottom of `findings.md` summarising:
- Pass / fail per criterion.
- Any new gotchas discovered during implementation or hardware verification (especially any that contradicted the spec's predictions).
- CPU measurement (peak + typical), detection latency post-playback, observed pass rate.
- Reference paths (spec + plan + code + deploy script).
Use the existing Test 1 and Test 2 outcome sections in `findings.md` as the template.
- [ ] **Step 10: Commit the findings update**
```bash
git add findings.md
git commit -m "Findings: Test 3 full-cycle probe outcome"
```
- [ ] **Step 11: Final status report**
Report to the user:
- Pass/fail per criterion (1 through 7 from spec).
- Any criteria that failed and what was discovered.
- A one-line summary of whether Test 3 unblocks the main Pi-client implementation.
---
## Self-review notes
- **Spec coverage:** every section of the spec maps to a task — Architecture/State machine/Components → Tasks 14. Why-Reset-matters → Task 2's `Reset()` + Task 3's PLAYBACK→IDLE branch. Error handling → Task 3's try/catch on the FireAndForget calls + `[status]` / `[error]` log lines. Gotchas-carried-forward → preserved verbatim in Task 3's code. Verification → Task 5.
- **Placeholder scan:** no TBD/TODO; every code step shows the actual code; every command shows expected output.
- **Type consistency:** `State` enum (`Idle`, `Beeping`, `Recording`, `Playback`) used consistently; `PlaybackDoneFlag.Set()` / `.Consume()` used consistently between `FireAndForgetPlayback` and the consumer's PLAYBACK branch; `WakewordModel.Reset()` called in two places (Recording-failure recovery and Playback→Idle), signature is parameterless in both.
@@ -0,0 +1,254 @@
# Test 3 — full-cycle C# probe (wakeword + record + playback)
Combine the C# Test 1 (record + play) and Test 2 (wakeword + beep) into a single state machine that exercises the full assistant conversation cycle on the Pi:
> Idle, listen for "alexa" → beep → record 5 s → play the recording back through the same USB Speaker Phone → idle, listen for "alexa" again. Repeat indefinitely until Ctrl-C.
This is the last throwaway probe before the main Pi-client code starts. Its job is to surface anything the prior two probes couldn't, in particular **whether the wakeword detector and the recording/playback path coexist correctly on one USB device with one persistent input stream**.
Read `findings.md` (§ "C# probe outcome", § "C# wakeword probe outcome") before reading this. Everything below assumes those two outcomes as ground truth.
## Goal
A single-binary C# probe that:
1. Opens one persistent PortAudio input stream and one `WakewordModel` for the lifetime of the process.
2. Runs a four-state machine on each incoming 80 ms audio frame: **IDLE → BEEPING → RECORDING → PLAYBACK → IDLE**.
3. Plays a 200 ms beep on detection (fire-and-forget output stream).
4. Records 5 s of audio into an in-memory `short[]` buffer.
5. Plays that buffer back through the same USB device (fire-and-forget output stream).
6. Resets the wakeword model on every IDLE re-entry so the next "alexa" is detected quickly (target ≤ ~2 s after playback ends; see "Why detector gating works"), not garbage from stale ring buffers.
7. Exits cleanly on Ctrl-C from the workstation (SIGINT propagated via `ssh -t`).
## Non-goals
- No automated tests. Verification is hardware-only (run on Pi, judge by ear + console output).
- No software AEC. The detector is gated during PLAYBACK (state-machine discards frames); that is sufficient to prevent the playback re-triggering the wakeword. The Anhui LISTENAI USB device may have hardware AEC; not in scope to probe.
- No custom wakeword. Stock "alexa" classifier carried forward from Test 2.
- No WAV output. Test 1 wrote `out.wav`; Test 3 keeps the recording in memory. If a future debugging session needs raw captures, add an opt-in flag then.
- No `WakewordModel` extraction into a shared library. The file is copied verbatim from Test 2 into Test 3, and `Reset()` is added only to the Test 3 copy. Test 2's copy is frozen.
- No barge-in (user interrupting playback by speaking). The detector is fully gated during PLAYBACK; there is no way for the user to retrigger mid-playback.
## Architecture
```
┌──────────────────────────────────────────┐
│ PortAudio input stream (persistent) │
│ 16 kHz mono Int16, 1280 frames/cb │
│ callback memcpy → BlockingCollection │
└────────────────┬─────────────────────────┘
│ short[1280] frames
┌──────────────────────────────────────────┐
│ Consumer thread │
│ while (!cts.IsCancellationRequested) { │
│ frame = queue.Take(cts.Token); │
│ switch (_state) { │
│ case IDLE → model.Predict(...) │
│ case BEEPING → discard + count │
│ case RECORDING→ memcpy → recordBuf │
│ case PLAYBACK → discard │
│ } │
│ } │
└────────────────┬─────────────────────────┘
│ on demand
┌──────────────┴──────────────┐
▼ ▼
FireAndForgetBeep FireAndForgetPlayback
(200 ms 880 Hz tone) (5.04 s of recordBuf)
Output stream pattern from Test 2 (Start under
try/catch BEFORE Task.Run + 2 s timeout on done.Wait)
```
One process. One persistent input stream. Per-cycle output streams spawned fire-and-forget. State machine lives entirely in the consumer thread, which is the single writer to `_state`, so `_state` only needs `volatile` — no locks.
## State machine
Frame size is 1280 samples = 80 ms at 16 kHz. All counters below are in frames.
| State | Per-frame action | Transition |
|-|-|-|
| **IDLE** | `score = model.Predict(frame)` | `score ≥ 0.5` AND cooldown elapsed → log `DETECTED`, call `FireAndForgetBeep(...)`, set `beepFrames = 0`, → **BEEPING** |
| **BEEPING** | discard frame; `beepFrames++` | after **3 frames** (240 ms) → allocate `recordBuf = new short[63 * 1280]`, `recordOffset = 0`, → **RECORDING** |
| **RECORDING** | `Buffer.MemoryCopy` frame into `recordBuf` at `recordOffset`; `recordOffset += 1280` | after **63 frames** (5.04 s, full buffer) → call `FireAndForgetPlayback(recordBuf, …)`, → **PLAYBACK** |
| **PLAYBACK** | discard frame; check `volatile bool _playbackDoneFlag` | `_playbackDoneFlag == true` → reset flag, call `model.Reset()`, → **IDLE** |
Rationale for the constants:
- **BEEPING = 3 frames (240 ms).** Beep duration is 200 ms; 3 × 80 ms = 240 ms is the next frame boundary at-or-above 200 ms, leaving a 40 ms margin to absorb device buffering, speaker reverb, and PortAudio's output latency so the recording doesn't open with the tail of the beep. If verification shows audible beep tail still leaking into the recording, raise to 4 frames (320 ms) — easy tweak.
- **RECORDING = 63 frames (5.04 s).** 5 s = 62.5 frames; rounding up to 63 avoids partial-frame handling. The extra 40 ms of recording is inaudibly different from 5 s.
- **Cooldown** applies only to the IDLE→BEEPING transition. The state machine itself blocks re-detection in the other states. 1 s cooldown carries forward from Test 2.
`_playbackDoneFlag` is set by the playback output stream's audio callback when its read offset reaches the end of `recordBuf`. The consumer checks the flag on every PLAYBACK frame (and only on PLAYBACK frames). Between the flag firing and the consumer seeing it, the consumer continues to drain input frames into the discard branch — that's what keeps the input stream from backing up.
## Why detector gating works (and why Reset matters)
The USB Speaker Phone is the same physical device for input and output. During PLAYBACK, the recorded audio comes out the speaker and re-enters the input stream. If `WakewordModel.Predict` were running across all states, any "alexa" buried in the recording would re-trigger detection, kicking the state machine back into BEEPING mid-playback — an infinite loop.
The state-machine gating (PLAYBACK discards every frame, doesn't call `Predict`) prevents the re-trigger directly. That much is straightforward.
The subtler issue is what happens to `WakewordModel`'s internal ring buffers (`_rawRing`, `_melRing`, `_embRing`) during BEEPING + RECORDING + PLAYBACK. With ~10 s of skipped `Predict` calls (240 ms BEEPING + 5.04 s RECORDING + ~5 s PLAYBACK), the rings hold audio from ~10 s ago — pre-detection. When we return to IDLE and resume `Predict`, the next call computes mel features from a window that straddles 10-second-old audio at the bottom and fresh post-playback audio at the top. The first one or two predictions on re-entry would be on this garbage window.
The `WarmupFrames = 16` guard already in `WakewordModel.cs` exists to suppress exactly this kind of cold-start junk on initial process boot. It does NOT re-arm after a pause. So without intervention, the first ~1.3 s after returning to IDLE would be a false-positive vulnerability window.
**Fix:** Add a `WakewordModel.Reset()` method that clears `_rawRingFill = 0`, `_melRing.Clear()`, `_embRing.Clear()`, `_framesSeen = 0`. The consumer calls it on the PLAYBACK→IDLE transition. The existing `WarmupFrames` guard then suppresses the next 16 `Predict` calls (~1.3 s of audio) until the rings refill with fresh post-playback audio. After that, detection runs as it did at startup.
This means the next "alexa" after playback can only be detected on a frame at least 1.3 s after the IDLE re-entry, plus whatever time the user takes to start saying "alexa". The verification budget for "next detection within ~1 s of playback" needs to acknowledge this: the practical lower bound is ~1.3 s (warmup) + ~0.5 s (user reaction + saying the word). See "Pass criteria" below for the adjusted threshold.
## Components and files
```
tests/03-full-cycle-cs/
├── Probe.csproj
│ <TargetFramework>net9.0</TargetFramework>
│ <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
│ <SelfContained>true</SelfContained>
│ <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
│ <InvariantGlobalization>true</InvariantGlobalization>
│ PackageReference PortAudioSharp2 1.0.6
│ PackageReference Microsoft.ML.OnnxRuntime 1.26.0
│ Content Include="models/*.onnx" CopyToOutputDirectory=PreserveNewest
│ RootNamespace=FullCycleProbe, AssemblyName=Probe
├── Program.cs ~200 lines, top-level statements:
│ 1. Environment.SetEnvironmentVariable + Libc.setenv for
│ PA_ALSA_PLUGHW=1 and ORT_LOGGING_LEVEL=3
│ 2. OrtEnv.CreateInstanceWithOptions with
│ logLevel = ORT_LOGGING_LEVEL_ERROR (BEFORE any InferenceSession)
│ 3. PortAudio.Initialize, FindUsbDevice (lifted from Test 2)
│ 4. WakewordModel ctor with models/{melspectrogram,embedding_model,alexa}.onnx
│ 5. BlockingCollection<short[]> queue (bounded 16)
│ 6. CancellationTokenSource + Console.CancelKeyPress handler
│ 7. Input StreamParameters + callback (memcpy → queue.TryAdd)
│ 8. Stream construction + Start
│ 9. Consumer loop with state-machine switch (sketched above)
│ 10. finally: stream.Stop, model.Dispose, PortAudio.Terminate
│ Helpers: FireAndForgetBeep (from Test 2), FireAndForgetPlayback (new,
│ same shape but reads recordBuf and sets _playbackDoneFlag),
│ MakeBeep, FindUsbDevice, static class Libc.setenv
├── WakewordModel.cs Copied verbatim from tests/02-wakeword-cs/.
│ Add Reset() method (≈5 lines):
│ public void Reset() {
│ _rawRingFill = 0;
│ _melRing.Clear();
│ _embRing.Clear();
│ _framesSeen = 0;
│ }
│ Namespace changes from WakewordProbe → FullCycleProbe.
└── models/ Copied from tests/02-wakeword-cs/models/.
melspectrogram.onnx, embedding_model.onnx, alexa.onnx
SHA-256 unchanged from the vendored set in commit 849ad47.
bin/probe-cs-3 Copy of bin/probe-cs-2 with two string swaps:
tests/02-wakeword-cs → tests/03-full-cycle-cs
~/probe-cs-2 → ~/probe-cs-3
PUBLISH_DIR /tmp/probe-cs-2-out → /tmp/probe-cs-3-out
ssh -t is retained (Ctrl-C propagation).
chmod +x at install time.
```
## Concurrency model
Three threads of execution in steady state:
1. **PortAudio input audio thread** (managed by the library). Runs the input callback every 80 ms. Allocates a `short[1280]`, memcpy's the input buffer, calls `queue.TryAdd(frame, 0)`. On `TryAdd` failure (queue full), logs `[status] consumer behind, dropping frame`. On `InputOverflow` status flag, logs `[status] input overflow`. Never blocks.
2. **Consumer thread** (`Task.Run` or just main thread after `stream.Start`). The state-machine loop above. Single writer of `_state`. Calls `model.Predict` in IDLE; calls `model.Reset` on PLAYBACK→IDLE.
3. **PortAudio output audio threads** (one per fire-and-forget beep/playback). Each output stream has its own callback that writes from a managed buffer (beep buffer or recordBuf). Sets `done.Set()` (beep) or `_playbackDoneFlag = true` (playback) when its source buffer is fully consumed.
Shared mutable state:
- `volatile State _state` — single writer (consumer), multiple readers (consumer only, actually). Volatile is enough.
- `volatile bool _playbackDoneFlag` — set by playback audio thread, read+reset by consumer.
- `BlockingCollection<short[]> queue` — its own concurrency.
No locks. No `Interlocked`. The state machine is intentionally a single-writer design so we don't have to reason about race conditions.
## Error handling
- **Input overflow** (PortAudio callback status flag): log `[status] input overflow`, continue. No corrective action — the buffer dropped audio at the kernel level, nothing the consumer can do.
- **Consumer-behind** (`queue.TryAdd` returns false because the bounded queue is full): log `[status] consumer behind, dropping frame`, continue. In steady state this should never fire; if it does during verification it's a signal that one of the state branches is too slow (likely `Predict` exceeding 80 ms, which Test 2 confirmed doesn't happen on the Pi).
- **Output stream `Start()` throws**: dispose the stream, log `[error] output stream failed to start: <ex>`, snap `_state` back to IDLE, call `model.Reset()`. The current cycle is lost; the probe survives. (Test 2 surfaced this as the exact failure mode the fire-and-forget pattern guards against.)
- **`WakewordModel` exceptions** during `Predict` or `Reset`: let propagate. The `finally` block disposes the model and terminates PortAudio. Probe exits non-zero. This is fail-fast on the assumption that any model exception is an invariant violation worth investigating.
- **Ctrl-C from workstation**: `Console.CancelKeyPress` calls `cts.Cancel()`. The consumer's `queue.Take(cts.Token)` throws `OperationCanceledException`, the outer `try` swallows it, the `finally` block runs `stream.Stop` + `model.Dispose` + `PortAudio.Terminate`. Process exits 0. `ssh -t` in `bin/probe-cs-3` ensures the SIGINT reaches the remote process.
## Gotchas carried forward (from findings.md)
1. **Libc.setenv for env vars libc reads via getenv().** `Environment.SetEnvironmentVariable` does NOT propagate to libc on Linux. Apply `Libc.setenv` for both `PA_ALSA_PLUGHW` (PortAudio) and `ORT_LOGGING_LEVEL` (onnxruntime) alongside the managed call.
2. **OrtEnv.CreateInstanceWithOptions BEFORE first InferenceSession.** ORT 1.26.0's GPU-detection logger fires from C++ before managed code runs, so `Libc.setenv("ORT_LOGGING_LEVEL", ...)` alone doesn't suppress the `/sys/class/drm/card0` warnings. Construct `OrtEnv` with `logLevel = ORT_LOGGING_LEVEL_ERROR` *before* `WakewordModel`'s constructor runs.
3. **`using Stream = PortAudioSharp.Stream;`** at the top of `Program.cs` to disambiguate from `System.IO.Stream` that `ImplicitUsings` pulls in.
4. **`<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`** in `Probe.csproj` for the callback's `Buffer.MemoryCopy` over `short*`.
5. **Fire-and-forget output stream lifetime.** `stream.Start()` runs synchronously under try/catch *before* `Task.Run` launches; on `Start()` failure, dispose + rethrow without ever creating the cleanup task. The cleanup task waits on `done.Wait(TimeSpan.FromSeconds(2))` — 2 s timeout so a faulting callback that silently fails (PortAudio swallows callback exceptions) can't hang cleanup. Applied to both `FireAndForgetBeep` AND `FireAndForgetPlayback`.
6. **Vendored alexa classifier has static batch dim `[1, 16, 96]`.** `WakewordModel`'s shape assertion intentionally skips dim 0 so both `1` (static) and `-1` (dynamic) pass. Preserve.
7. **Mel-spectrogram output shape `(1, 1, n_frames, 32)`.** `n_frames` lives at `Dimensions[2]`, not `[1]`. Preserve in the copied `WakewordModel.cs`.
8. **`ssh -t` for Ctrl-C propagation.** `bin/probe-cs-3` forces a pseudo-TTY so workstation SIGINT reaches the remote `./Probe` and triggers `Console.CancelKeyPress`.
## Verification (hardware only, no automated tests)
Run from workstation:
```
./bin/probe-cs-3
```
Expected console flow (abridged):
```
[wakeword-model] melspectrogram.onnx
input 'input': shape=[1,-1] dtype=Single
output 'output': shape=[1,1,-1,32] dtype=Single
[wakeword-model] embedding_model.onnx
...
[wakeword-model] alexa.onnx
input 'onnx::Unsqueeze_0': shape=[1,16,96] dtype=Single
output ...
Loaded in 0.6s.
Using device 3 ('USB Speaker Phone (LISTENAI ...)')
Listening. Say 'alexa'. Ctrl-C to exit.
DETECTED alexa score=0.872 t=4.3s
[state] BEEPING
[state] RECORDING
[state] PLAYBACK
[state] IDLE
DETECTED alexa score=0.811 t=15.2s
...
```
(Exact `[wakeword-model]` shape output reflects whatever the ONNX models actually report; the example above is illustrative.)
### Pass criteria
All must hold during a single run on the Pi:
1. **3 full cycles back-to-back** without restarting the probe. Each playback is clearly recognisable as what was spoken into the mic in the preceding 5 s.
2. **Next "alexa" after playback** is detected within ~2 s of playback ending. (The ~1.3 s `WarmupFrames` re-arm after `model.Reset()` is a hard floor; budget another ~0.5 s for user reaction + saying the word. If detection consistently takes longer than ~2 s, that's a regression vs. Test 2's IDLE-state detection latency and worth investigating.)
3. **Wakeword detection rate during IDLE** ≥ 8/10 attempts (carries forward from Test 2's measured rate).
4. **CPU during IDLE steady state** < 50% of one core, measured by `top` on the Pi. (Test 2 measured 1731% on the same workload; Test 3 adds only a state-machine switch — no extra CPU in IDLE.)
5. **No `[status] input overflow` or `[status] consumer behind` warnings** during any phase of normal operation.
6. **No stuck states.** The probe never sits in BEEPING, RECORDING, or PLAYBACK for longer than its bounded duration. (BEEPING ≤ 240 ms, RECORDING ≤ 5.04 s, PLAYBACK ≤ ~5.5 s including device drain.)
7. **Workstation Ctrl-C** triggers a clean exit: `\nbye` line, process exit 0, no stack trace, terminal returns to prompt.
### What success teaches us for the main assistant
- The "persistent input stream + state-machine consumer + per-cycle output streams" shape works on this hardware. The main assistant can build directly on it.
- Detector gating during playback (without AEC) is sufficient to prevent self-trigger, at least with stock "alexa" and the recorded user audio. Real TTS may need re-evaluation; this probe doesn't answer that.
- `WakewordModel.Reset()` belongs in the production model interface — the main assistant will need it for the same reasons.
### What failure would tell us
- If detection rate drops below 8/10 in IDLE: the state-machine overhead is interfering. Investigate (unlikely — the switch is O(constant)).
- If `[status] consumer behind` warnings appear: one of the consumer branches is too slow. Most likely `Predict` exceeded 80 ms (Test 2 didn't measure this directly). Add per-`Predict` timing and re-run.
- If playback re-triggers detection (probe loops in BEEPING/RECORDING/PLAYBACK indefinitely): `Reset` is wrong, or `_playbackDoneFlag` plumbing has a race. Investigate via logging which state the trigger occurred in.
- If `Ctrl-C` produces a stack trace: `OperationCanceledException` isn't being caught at the right level. Fix.
## Reference paths
- Spec (this doc): `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`
- Plan (next): `docs/superpowers/plans/2026-06-12-test-3-full-cycle-csharp.md`
- Code (to be written): `tests/03-full-cycle-cs/`
- Deploy (to be written): `bin/probe-cs-3`
- Test 1 reference: `tests/01-record-play-cs/Program.cs`, `tests/01-record-play-cs/WavWriter.cs`, `bin/probe-cs`
- Test 2 reference: `tests/02-wakeword-cs/Program.cs`, `tests/02-wakeword-cs/WakewordModel.cs`, `bin/probe-cs-2`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
- Canonical findings: `findings.md` § "C# probe outcome (2026-06-12 — Test 1 ported to C#)" and § "C# wakeword probe outcome (2026-06-12 — Test 2 ported to C#)"
+33
View File
@@ -211,6 +211,39 @@ The Python Test 2 (openwakeword "alexa" listener with beep on detect) was re-imp
- Code: `tests/02-wakeword-cs/`
- Deploy: `bin/probe-cs-2`
## C# full-cycle outcome (2026-06-12 — Test 3)
Test 3 stitched the C# Test 1 (record + play) and Test 2 (wakeword + beep) into one state machine: idle → "alexa" → beep → record 5 s → playback → idle. Probe lives at `tests/03-full-cycle-cs/`; deploy script `bin/probe-cs-3`. All seven spec pass criteria held on hardware on the first run-after-fix (see "Key gotchas" below).
### What we proved
- **The single-input-stream + state-machine + per-cycle-output-streams shape works on this hardware.** One PortAudio input stream open for the process lifetime feeds a `BlockingCollection<short[]>` (bounded 16); a single consumer thread dispatches each 80 ms frame through IDLE / BEEPING / RECORDING / PLAYBACK. No locks, no `Interlocked` — `state` is consumer-thread-local; the only cross-thread datum is a `volatile bool` in a `PlaybackDoneFlag`.
- **Detector gating during PLAYBACK prevents self-trigger.** The recorded user audio comes back out the speaker and re-enters the input stream; because the consumer's PLAYBACK branch discards frames (never calls `model.Predict`), the playback never re-triggers detection.
- **`WakewordModel.Reset()` on PLAYBACK→IDLE re-arms the warmup guard correctly.** Five lines (`_rawRingFill = 0; _melRing.Clear(); _embRing.Clear(); _framesSeen = 0`) restore the model to the same state it has just after construction; the existing `WarmupFrames = 16` then suppresses the next ~1.3 s of `Predict` calls until the rings refill with fresh post-playback audio. Next "alexa" after playback ends is detected reliably and quickly (within target).
- **3+ cycles back-to-back, no degradation.** Detection rate holds, CPU stays in IDLE-state range between cycles, no stuck states.
- **No `[status]` or `[error]` warnings during normal operation.** Input doesn't overflow during PLAYBACK (the consumer keeps draining the queue into the discard branch), consumer doesn't fall behind, no fire-and-forget timeouts after the fix below.
- **Ctrl-C from the workstation cleanly exits the probe.** Same `ssh -t` + `Console.CancelKeyPress` + `cts.Cancel()` pattern as Test 2.
### Key gotchas
- **The fire-and-forget cleanup task's `done.Wait` timeout must scale to the buffer length, not be a flat constant.** Test 2's `FireAndForgetBeep` used `done.Wait(TimeSpan.FromSeconds(2))` to guard against silently-faulted PortAudio callbacks (PortAudio swallows callback exceptions). For a 200 ms beep, 2 s is generous; the cleanup task's wait always returns "set", then sleeps 200 ms and disposes. Test 3's `FireAndForgetPlayback` initially inherited the same flat 2 s timeout — but the recording playback runs for ~5 s, so the timeout fired mid-playback, the cleanup task called `stream.Stop()` 200 ms later, and the audio cut at ~2.2 s. **First hardware run produced "[error] recording playback timed out — callback may have faulted" on every cycle.** Fix: compute `expectedSeconds = buf.Length / SampleRate` and use `TimeSpan.FromSeconds(expectedSeconds + 2.0)` as the cleanup wait. Commit `a7d408e`.
- **Partial deploys are silently wrong: a self-contained .NET publish uses `Probe` (native launcher) + `Probe.dll` (managed code).** Copying only `Probe` after a rebuild leaves the old `.dll` on the target, which the new launcher loads — so the binary on the Pi reflects code from the previous build. Always wipe-and-replace the whole publish directory (the pattern already in `bin/probe-cs-3`: `rm -rf ~/probe-cs-3 && mkdir ~/probe-cs-3 && scp -r $PUBLISH_DIR/. pi:~/probe-cs-3/`). Don't do incremental `scp Probe`.
- **`PlaybackDoneFlag` should be created per cycle, not reused.** A theoretical late `Set()` from a delayed cleanup `Task.Run` (cycle N) could land on the shared flag while the consumer is in cycle N+1's PLAYBACK, causing a spurious immediate PLAYBACK→IDLE that skips playback. Allocating a fresh `PlaybackDoneFlag()` inside the RECORDING→PLAYBACK transition (and reading with `playbackDone!.Consume()` in PLAYBACK) sends the stale `Set()` to a now-unreachable object. Commit `070cc07`. Not observed in practice but cheap to defend.
- **Log the cleanup-task's `done.Wait` timeout return.** Without the log, a silently-faulted PortAudio callback produces "PLAYBACK→IDLE with no audio" with no diagnostic — exactly the failure mode the timeout was supposed to defend against. Both `FireAndForgetBeep` and `FireAndForgetPlayback` now log `[error] ... timed out — callback may have faulted` on `!done.Wait(timeout)`. Commit `070cc07`.
### Implications for the main assistant
- The whole shape — persistent input stream, single-consumer state machine, per-cycle output streams, fire-and-forget cleanup with timeout scaled to buffer length — is good to lift directly into the main Pi client. The main assistant adds: Opus encoding on the recorded buffer before sending to the Realtime WebSocket, an Opus decoder feeding the output stream during TTS playback, and replacing the local 5 s recording window with VAD-driven endpointing. None of those replace the state-machine shape; they slot in.
- `WakewordModel` should be promoted out of `tests/` into a real source tree (e.g. `src/Audio/Wakeword/`) when the main assistant starts. Test 3's copy is the live version (it has `Reset()`); Test 2's is frozen.
- The "always wipe-and-replace the whole publish directory" deploy pattern needs to be the default in the main assistant's deploy script too. A partial `scp` of `Probe` is a footgun every time the .dll changes (i.e. every code change).
### Reference paths (C# full-cycle probe)
- Spec: `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`
- Plan: `docs/superpowers/plans/2026-06-12-test-3-full-cycle-csharp.md`
- Code: `tests/03-full-cycle-cs/`
- Deploy: `bin/probe-cs-3`
## Reference paths
- Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
+15 -1
View File
@@ -59,8 +59,11 @@ internal sealed class WakewordModel : IDisposable
};
_mel = new InferenceSession(melPath, opts);
PrintShapes(Path.GetFileName(melPath), _mel);
_emb = new InferenceSession(embeddingPath, opts);
PrintShapes(Path.GetFileName(embeddingPath), _emb);
_cls = new InferenceSession(classifierPath, opts);
PrintShapes(Path.GetFileName(classifierPath), _cls);
// Discover input names + assert shape geometry.
_melInputName = _mel.InputMetadata.Keys.Single();
@@ -70,6 +73,17 @@ internal sealed class WakewordModel : IDisposable
AssertClassifierShape(_cls); // [batch, 16, 96], dtype float
}
private static void PrintShapes(string filename, InferenceSession sess)
{
Console.Error.WriteLine($"[wakeword-model] {filename}");
foreach (var kv in sess.InputMetadata)
Console.Error.WriteLine(
$" input '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
foreach (var kv in sess.OutputMetadata)
Console.Error.WriteLine(
$" output '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
}
private static void AssertEmbeddingShape(InferenceSession sess)
{
if (!sess.InputMetadata.TryGetValue(EmbeddingInputName, out var meta))
@@ -144,7 +158,7 @@ internal sealed class WakewordModel : IDisposable
var melOut = melResults.First().AsTensor<float>(); // shape (1, 1, n_frames, 32)
// Apply openwakeword's `x / 10 + 2` transform and append each new frame to _melRing.
// FIXED: actual mel output shape is (1, 1, n_frames, 32) n_frames is at Dimensions[2], NOT [1].
// Mel output shape is (1, 1, n_frames, 32). Note n_frames is at Dimensions[2], not [1].
int nFrames = melOut.Dimensions[2];
for (int f = 0; f < nFrames; f++)
{
+24
View File
@@ -0,0 +1,24 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>FullCycleProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.26.0" />
</ItemGroup>
<ItemGroup>
<Content Include="models/*.onnx">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project>
+387
View File
@@ -0,0 +1,387 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Runtime.InteropServices;
using FullCycleProbe;
using Microsoft.ML.OnnxRuntime;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
const int SampleRate = WakewordModel.SampleRate; // 16 000
const int Channels = 1;
const uint BlockFrames = WakewordModel.FrameSamples; // 1280 (80 ms)
const float Threshold = 0.5f;
const double CooldownSeconds = 1.0;
const double BeepHz = 880.0;
const double BeepSeconds = 0.2;
const int BeepingFrames = 3; // 3 * 80 ms = 240 ms (covers 200 ms beep + 40 ms drain/reverb margin)
const int RecordingFrames = 63; // 63 * 80 ms = 5.04 s (rounded up from 5 s to avoid partial-frame handling)
const int RecordBufSamples = RecordingFrames * (int)BlockFrames; // 80640
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio / onnxruntime (which use getenv()) see the values.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
Environment.SetEnvironmentVariable("ORT_LOGGING_LEVEL", "3");
Libc.setenv("ORT_LOGGING_LEVEL", "3", 1);
// ORT 1.26.0: the env var does NOT suppress early GPU-discovery warnings from device_discovery.cc.
// CreateInstanceWithOptions sets the log level at OrtEnv creation time, before EP discovery runs.
var ortEnvOpts = new EnvironmentCreationOptions { logId = "FullCycleProbe", logLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR };
OrtEnv.CreateInstanceWithOptions(ref ortEnvOpts);
PortAudio.Initialize();
WakewordModel? model = null;
try
{
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
Console.WriteLine("Loading WakewordModel...");
var t0 = DateTime.UtcNow;
string modelsDir = Path.Combine(AppContext.BaseDirectory, "models");
model = new WakewordModel(
Path.Combine(modelsDir, "melspectrogram.onnx"),
Path.Combine(modelsDir, "embedding_model.onnx"),
Path.Combine(modelsDir, "alexa.onnx"));
Console.WriteLine($"Loaded in {(DateTime.UtcNow - t0).TotalSeconds:0.00}s.");
var queue = new BlockingCollection<short[]>(boundedCapacity: 16);
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var inParams = new StreamParameters
{
device = device,
channelCount = Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (frameCount != BlockFrames)
{
Console.Error.WriteLine($"[status] unexpected callback frameCount={frameCount} (want {BlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[BlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
if (!queue.TryAdd(frame, 0))
Console.Error.WriteLine("[status] consumer behind, dropping frame");
return StreamCallbackResult.Continue;
};
using var stream = new Stream(
inParams, null, SampleRate, BlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
stream.Start();
short[] beepBuffer = MakeBeep(BeepHz, BeepSeconds, SampleRate);
Console.WriteLine("Listening. Say 'alexa'. Ctrl-C to exit.");
var sw = Stopwatch.StartNew();
TimeSpan lastTrigger = TimeSpan.FromSeconds(-CooldownSeconds);
// State machine (single-writer: this consumer thread).
State state = State.Idle;
int beepFrames = 0;
int recordOffset = 0;
short[] recordBuf = null!; // assigned on BEEPING→RECORDING transition
PlaybackDoneFlag? playbackDone = null;
try
{
while (!cts.IsCancellationRequested)
{
short[] frame = queue.Take(cts.Token);
switch (state)
{
case State.Idle:
{
float score = model.Predict(frame);
var now = sw.Elapsed;
if (score >= Threshold && (now - lastTrigger).TotalSeconds >= CooldownSeconds)
{
Console.WriteLine($"DETECTED alexa score={score:0.000} t={now.TotalSeconds:0.0}s");
lastTrigger = now;
try
{
FireAndForgetBeep(device, beepBuffer);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] beep stream failed to start: {ex.Message}");
// Stay in IDLE — no cycle to abort yet.
break;
}
Console.WriteLine("[state] BEEPING");
state = State.Beeping;
beepFrames = 0;
}
break;
}
case State.Beeping:
{
beepFrames++;
if (beepFrames >= BeepingFrames)
{
Console.WriteLine("[state] RECORDING");
state = State.Recording;
recordBuf = new short[RecordBufSamples];
recordOffset = 0;
}
break;
}
case State.Recording:
{
unsafe
{
fixed (short* src = &frame[0])
fixed (short* dst = &recordBuf[recordOffset])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
recordOffset += (int)BlockFrames;
if (recordOffset >= RecordBufSamples)
{
try
{
playbackDone = new PlaybackDoneFlag();
FireAndForgetPlayback(device, recordBuf, playbackDone);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] playback stream failed to start: {ex.Message}");
// Drop the recording, reset model, return to IDLE.
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
break;
}
Console.WriteLine("[state] PLAYBACK");
state = State.Playback;
}
break;
}
case State.Playback:
{
// Discard input during playback (gate the detector).
if (playbackDone!.Consume())
{
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
}
break;
}
}
}
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
stream.Stop();
Console.WriteLine("\nbye");
}
finally
{
model?.Dispose();
PortAudio.Terminate();
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static void FireAndForgetBeep(int device, short[] beepBuffer)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = beepBuffer.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &beepBuffer[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= beepBuffer.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
Task.Run(() =>
{
// 2 s timeout so a failed callback (which PortAudio silently swallows
// and would leave 'done' unset) eventually releases the stream + handle
// instead of leaking them for the process lifetime.
if (!done.Wait(TimeSpan.FromSeconds(2)))
Console.Error.WriteLine("[error] beep playback timed out — callback may have faulted");
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
});
}
static void FireAndForgetPlayback(int device, short[] recordBuf, PlaybackDoneFlag doneFlag)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = recordBuf.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &recordBuf[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= recordBuf.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
// Timeout = playback wall-clock + 2 s safety margin. Test 2's beep cleanup uses
// a flat 2 s because beep is 200 ms; for a 5 s recording the same value would
// fire mid-playback, Stop the stream, and cut the audio.
double expectedSeconds = recordBuf.Length / (double)WakewordModel.SampleRate;
var waitTimeout = TimeSpan.FromSeconds(expectedSeconds + 2.0);
Task.Run(() =>
{
if (!done.Wait(waitTimeout))
Console.Error.WriteLine("[error] recording playback timed out — callback may have faulted");
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
doneFlag.Set(); // tell the consumer it's safe to flip PLAYBACK→IDLE
});
}
enum State { Idle, Beeping, Recording, Playback }
// Volatile flag set by the playback cleanup task and consumed by the consumer thread.
// Wrapped in a tiny class so we can pass it by reference into the helper.
sealed class PlaybackDoneFlag
{
private volatile bool _set;
public void Set() => _set = true;
public bool Consume()
{
if (!_set) return false;
_set = false;
return true;
}
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}
+226
View File
@@ -0,0 +1,226 @@
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
namespace FullCycleProbe;
// Streaming wakeword inference port of openwakeword 0.6.0's predict pipeline.
// References (read alongside this file):
// openwakeword/utils.py:AudioFeatures._streaming_features (mel + embedding stages)
// openwakeword/utils.py:AudioFeatures._streaming_melspectrogram
// openwakeword/utils.py:AudioFeatures._get_embeddings
// openwakeword/model.py:Model.predict (classifier stage)
internal sealed class WakewordModel : IDisposable
{
// Audio I/O geometry — per-call input contract.
public const int FrameSamples = 1280; // 80 ms @ 16 kHz; each Predict() call.
public const int SampleRate = 16_000;
// Mel model: input is last (FrameSamples + MelContextSamples) raw samples
// -- the +480 is openwakeword's `-n_samples-160*3:` slice (3 hops of context).
public const int MelContextSamples = 480; // 160 * 3
public const int MelInputSamples = FrameSamples + MelContextSamples; // 1760
public const int MelBins = 32; // melspectrogram model output dim
public const int MelBufferMaxFrames = 970; // 10 * 97 (openwakeword `melspectrogram_max_len`)
// Embedding model: 76-mel-frame window in, 96-d embedding out.
public const int EmbeddingWindowMelFrames = 76;
public const int EmbeddingDim = 96;
public const int EmbeddingBufferMax = 120; // openwakeword `feature_buffer_max_len`
public const string EmbeddingInputName = "input_1"; // openwakeword convention; assert at startup
// Classifier: 16 embeddings in, scalar score out.
public const int ClassifierEmbeddings = 16;
// Skip the first N Predict() calls — buffer fill-up window.
public const int WarmupFrames = ClassifierEmbeddings; // 16 frames ≈ 1.28 s
private readonly InferenceSession _mel;
private readonly InferenceSession _emb;
private readonly InferenceSession _cls;
private readonly string _melInputName; // discovered at startup
private readonly string _clsInputName; // discovered at startup
private readonly short[] _rawRing = new short[MelInputSamples];
private int _rawRingFill = 0; // samples buffered (≤ MelInputSamples)
private readonly List<float[]> _melRing = new(MelBufferMaxFrames); // each entry is a length-32 mel frame
private readonly List<float[]> _embRing = new(EmbeddingBufferMax); // each entry is a length-96 embedding
private int _framesSeen = 0;
public WakewordModel(string melPath, string embeddingPath, string classifierPath)
{
var opts = new SessionOptions
{
IntraOpNumThreads = 1,
InterOpNumThreads = 1,
LogSeverityLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR,
};
_mel = new InferenceSession(melPath, opts);
PrintShapes(Path.GetFileName(melPath), _mel);
_emb = new InferenceSession(embeddingPath, opts);
PrintShapes(Path.GetFileName(embeddingPath), _emb);
_cls = new InferenceSession(classifierPath, opts);
PrintShapes(Path.GetFileName(classifierPath), _cls);
// Discover input names + assert shape geometry.
_melInputName = _mel.InputMetadata.Keys.Single();
AssertEmbeddingShape(_emb); // input_1: [batch, 76, 32, 1], dtype float
_clsInputName = _cls.InputMetadata.Keys.Single();
AssertClassifierShape(_cls); // [batch, 16, 96], dtype float
}
private static void PrintShapes(string filename, InferenceSession sess)
{
Console.Error.WriteLine($"[wakeword-model] {filename}");
foreach (var kv in sess.InputMetadata)
Console.Error.WriteLine(
$" input '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
foreach (var kv in sess.OutputMetadata)
Console.Error.WriteLine(
$" output '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
}
private static void AssertEmbeddingShape(InferenceSession sess)
{
if (!sess.InputMetadata.TryGetValue(EmbeddingInputName, out var meta))
throw new InvalidOperationException(
$"embedding_model.onnx: expected input named '{EmbeddingInputName}', got [{string.Join(",", sess.InputMetadata.Keys)}]");
var d = meta.Dimensions;
// Expected: [batch, 76, 32, 1] — batch may be -1 (dynamic).
if (d.Length != 4 || d[1] != EmbeddingWindowMelFrames || d[2] != MelBins || d[3] != 1)
throw new InvalidOperationException(
$"embedding_model.onnx: expected input shape [batch,{EmbeddingWindowMelFrames},{MelBins},1], got [{string.Join(",", d)}]");
if (meta.ElementType != typeof(float))
throw new InvalidOperationException($"embedding_model.onnx: expected Single input, got {meta.ElementType.Name}");
}
private static void AssertClassifierShape(InferenceSession sess)
{
var inputName = sess.InputMetadata.Keys.Single();
var meta = sess.InputMetadata[inputName];
var d = meta.Dimensions;
// Expected: [batch, 16, 96] — batch may be -1.
if (d.Length != 3 || d[1] != ClassifierEmbeddings || d[2] != EmbeddingDim)
throw new InvalidOperationException(
$"alexa.onnx: expected input shape [batch,{ClassifierEmbeddings},{EmbeddingDim}], got [{string.Join(",", d)}]");
if (meta.ElementType != typeof(float))
throw new InvalidOperationException($"alexa.onnx: expected Single input, got {meta.ElementType.Name}");
}
public float Predict(short[] frame1280)
{
if (frame1280.Length != FrameSamples)
throw new ArgumentException($"Expected {FrameSamples} samples, got {frame1280.Length}");
// 1. Append 1280 new samples to the raw ring (shift older samples down if full).
if (_rawRingFill < MelInputSamples)
{
int copyToFront = Math.Min(MelInputSamples - _rawRingFill, FrameSamples);
Array.Copy(frame1280, 0, _rawRing, _rawRingFill, copyToFront);
_rawRingFill += copyToFront;
if (copyToFront < FrameSamples)
{
// Boundary case: ring was partially full and the new frame overshoots
// remaining capacity. Fires exactly once during warm-up (typically call 2,
// when _rawRingFill = 1280 and the incoming 1280 samples overshoot the
// remaining 480 capacity). Shift the older samples left to make room,
// then write the leftover at the tail.
int leftover = FrameSamples - copyToFront;
Array.Copy(_rawRing, leftover, _rawRing, 0, MelInputSamples - leftover);
Array.Copy(frame1280, copyToFront, _rawRing, MelInputSamples - leftover, leftover);
}
}
else
{
// Shift older samples left by FrameSamples, then append new at the tail.
Array.Copy(_rawRing, FrameSamples, _rawRing, 0, MelInputSamples - FrameSamples);
Array.Copy(frame1280, 0, _rawRing, MelInputSamples - FrameSamples, FrameSamples);
}
// Skip everything until we have the full mel-context window primed.
if (_rawRingFill < MelInputSamples)
{
_framesSeen++;
return 0f;
}
// 2. Mel stage: feed the full _rawRing as float32 (1, MelInputSamples) into mel model.
var melInputData = new float[MelInputSamples];
for (int i = 0; i < MelInputSamples; i++) melInputData[i] = _rawRing[i]; // int16 → float32, NO normalisation
var melInputTensor = new DenseTensor<float>(melInputData, new[] { 1, MelInputSamples });
using var melResults = _mel.Run(new[] {
NamedOnnxValue.CreateFromTensor(_melInputName, melInputTensor)
});
var melOut = melResults.First().AsTensor<float>(); // shape (1, 1, n_frames, 32)
// Apply openwakeword's `x / 10 + 2` transform and append each new frame to _melRing.
// Mel output shape is (1, 1, n_frames, 32). Note n_frames is at Dimensions[2], not [1].
int nFrames = melOut.Dimensions[2];
for (int f = 0; f < nFrames; f++)
{
var bin = new float[MelBins];
for (int b = 0; b < MelBins; b++)
bin[b] = melOut[0, 0, f, b] / 10f + 2f;
_melRing.Add(bin);
}
if (_melRing.Count > MelBufferMaxFrames)
_melRing.RemoveRange(0, _melRing.Count - MelBufferMaxFrames);
// 3. Embedding stage: need ≥ 76 mel frames; slice the last 76 → (1, 76, 32, 1).
if (_melRing.Count < EmbeddingWindowMelFrames)
{
_framesSeen++;
return 0f;
}
var embInputData = new float[EmbeddingWindowMelFrames * MelBins];
int startMel = _melRing.Count - EmbeddingWindowMelFrames;
for (int f = 0; f < EmbeddingWindowMelFrames; f++)
Array.Copy(_melRing[startMel + f], 0, embInputData, f * MelBins, MelBins);
var embInputTensor = new DenseTensor<float>(embInputData, new[] { 1, EmbeddingWindowMelFrames, MelBins, 1 });
using var embResults = _emb.Run(new[] {
NamedOnnxValue.CreateFromTensor(EmbeddingInputName, embInputTensor)
});
var embOut = embResults.First().AsTensor<float>(); // shape (1, 1, 1, 96)
var newEmb = new float[EmbeddingDim];
for (int i = 0; i < EmbeddingDim; i++) newEmb[i] = embOut[0, 0, 0, i];
_embRing.Add(newEmb);
if (_embRing.Count > EmbeddingBufferMax)
_embRing.RemoveRange(0, _embRing.Count - EmbeddingBufferMax);
_framesSeen++;
// 4. Warm-up + classifier stage.
if (_embRing.Count < ClassifierEmbeddings || _framesSeen <= WarmupFrames)
return 0f;
var clsInputData = new float[ClassifierEmbeddings * EmbeddingDim];
int startEmb = _embRing.Count - ClassifierEmbeddings;
for (int i = 0; i < ClassifierEmbeddings; i++)
Array.Copy(_embRing[startEmb + i], 0, clsInputData, i * EmbeddingDim, EmbeddingDim);
var clsInputTensor = new DenseTensor<float>(clsInputData, new[] { 1, ClassifierEmbeddings, EmbeddingDim });
using var clsResults = _cls.Run(new[] {
NamedOnnxValue.CreateFromTensor(_clsInputName, clsInputTensor)
});
var clsOut = clsResults.First().AsTensor<float>(); // shape (1, 1)
return clsOut[0, 0];
}
public void Reset()
{
_rawRingFill = 0;
_melRing.Clear();
_embRing.Clear();
_framesSeen = 0;
}
public void Dispose()
{
_mel.Dispose();
_emb.Dispose();
_cls.Dispose();
}
}
Binary file not shown.
Binary file not shown.
Binary file not shown.