Compare commits

...

18 Commits

Author SHA1 Message Date
tes cd07b52f8f Findings: Test 4a OpenAI Realtime one-shot voice round-trip outcome 2026-06-12 19:42:35 +00:00
tes 4cfb0d4b9a Test 4a: migrate session.update + event names to GA Realtime API
Hardware verification surfaced four shape changes from the beta API that
the spec/plan was written against:

1. Remove the OpenAI-Beta: realtime=v1 header (GA rejects it with
   beta_api_shape_disabled).
2. session.update payload nests under session.audio.{input,output} instead
   of flat fields. session.type = "realtime" + session.model are inline.
3. session.audio.{input,output}.format is now an object
   ({ type: "audio/pcm", rate: 24000 }), not a bare "pcm16" string.
4. 'session.modalities' is no longer accepted; dropped (server uses defaults).
5. response.audio.*  →  response.output_audio.*  (delta, done, transcript).

Also silenced a bunch of GA framing events (conversation.item.added /
.done, response.output_item.added / .done, response.content_part.added /
.done, rate_limits.updated) that were burying signal in the [ws] ignored
log lines.

Plus added an outer try/catch around main so a mid-startup WS error event
(which fires cts.Cancel via the receive loop) routes through cleanup
instead of crashing on the await SendSessionUpdate.

Verified on hardware: full round-trip works, t=0.44 s end-of-speech to
first audio sample arriving, assistant reply audible, clean exit.
2026-06-12 19:41:08 +00:00
tes ac8a3f012c Test 4a: bin/probe-cs-4a deploy script with ~/.openai_key precondition 2026-06-12 19:16:32 +00:00
tes a590c96d07 Test 4a: wire NoMoreDeltas + add UpstreamSendLoop stop-and-empty exit
Code-review feedback from Task 2:
- Use state.NoMoreDeltas in the output callback completion check instead
  of downstream.Reader.Completion.IsCompleted (spec called for it as the
  redundant fast path; it was dead code before).
- UpstreamSendLoop now break's when StopSending && upstream channel empty,
  matching the spec's stated exit shape.
2026-06-12 19:15:29 +00:00
tes a1a3b5c981 Test 4a: WS plumbing + audio streams + server VAD round-trip 2026-06-12 19:09:26 +00:00
tes 43b3a1a0f1 Test 4a scaffold: csproj for tests/04a-realtime-oneshot-cs 2026-06-12 19:00:57 +00:00
tes e96b6ffd95 Test 4a: implementation plan
Five tasks: scaffold csproj → write full Program.cs (WS + audio
streams + channels + server VAD round-trip) → bin/probe-cs-4a with
~/.openai_key precondition → one-time Pi key setup → hardware
verification against the spec's 8 pass criteria.
2026-06-12 17:34:25 +00:00
tes dea126f994 Test 4a: spec — OpenAI Realtime API one-shot voice round-trip
Boot → beep → user speaks → server VAD ends turn → assistant audio
plays back → exit 0. No wakeword, no loop, no state machine.

Splits Test 4 into 4a (this spec) and 4b (wakeword + loop + 16↔24 kHz
coexistence, future). 4a debugs the OpenAI Realtime surface — auth, WS
handshake, session.update payload, server VAD, base64 audio, streaming-
length output-stream cleanup — in isolation from 4b's harder concurrency.
2026-06-12 16:59:10 +00:00
tes cbb47d1465 Findings: Test 3 C# full-cycle probe outcome 2026-06-12 13:56:16 +00:00
tes a7d408e903 Test 3: scale playback timeout to buffer length (was hardcoded 2 s)
FireAndForgetPlayback's cleanup task had the same 2 s done.Wait timeout
as the beep cleanup. Beep is 200 ms so 2 s is generous; recording
playback is 5 s so the timeout fired mid-playback, cleanup ran
stream.Stop(), and the audio cut at ~2.2 s. Timeout is now
recording-duration + 2 s.
2026-06-12 13:54:16 +00:00
tes 1d9a417cfc Test 3: bin/probe-cs-3 deploy script 2026-06-12 13:40:55 +00:00
tes 070cc0767a Test 3: per-cycle PlaybackDoneFlag + log timeout in fire-and-forget cleanup
Stale Set() from a delayed cleanup task could short-circuit a future
PLAYBACK state; new flag per cycle avoids the cross-cycle leak. Also
log when the 2 s done.Wait timeout fires so silent PortAudio callback
faults are visible during hardware debugging.
2026-06-12 13:38:09 +00:00
tes 5a498e9c91 Test 3: full-cycle state machine (IDLE/BEEPING/RECORDING/PLAYBACK) 2026-06-12 13:31:25 +00:00
tes 222d58baeb Test 3: copy WakewordModel + models from Test 2, add Reset() 2026-06-12 13:25:40 +00:00
tes 6fa2779ea1 Test 3 scaffold: csproj for tests/03-full-cycle-cs 2026-06-12 13:22:23 +00:00
tes 14690484de Test 3 plan: implementation steps for full-cycle C# probe
Five tasks: scaffold csproj, copy WakewordModel + models with Reset(),
write Program.cs with the four-state machine, write bin/probe-cs-3
deploy script, and hardware verification on the Pi.
2026-06-12 13:19:18 +00:00
tes e9730c7e50 Test 3 spec: full-cycle C# probe (wakeword + record + playback)
State machine on a single persistent input stream: IDLE → BEEPING →
RECORDING → PLAYBACK → IDLE. Detector gated during non-IDLE states;
WakewordModel.Reset() called on IDLE re-entry to clear stale ring
buffers and re-arm the warmup guard.
2026-06-12 13:13:59 +00:00
tes b6d9f9a1cf WakewordModel: print shapes at startup + tidy mel-shape comment
Add PrintShapes() helper that writes input/output name, shape, and dtype
for each InferenceSession to stderr at construction time, prefixed with
[wakeword-model]. Called after each of the three session creations so the
geometry is visible on first run without grep-ing logs.

Also rephrase the stale "FIXED:" comment on the mel Dimensions[2] indexing
to plain explanatory text that describes the current invariant.
2026-06-12 12:19:59 +00:00
16 changed files with 3758 additions and 1 deletions
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-3-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/03-full-cycle-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-3/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-3 && mkdir ~/probe-cs-3'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-3/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-3 && chmod +x Probe && ./Probe'
+36
View File
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-4a-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> precondition: ~/.openai_key present on Pi"
if ! sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'test -f ~/.openai_key && test -r ~/.openai_key'; then
echo "ERROR: ~/.openai_key missing or unreadable on Pi."
echo "Create it on the Pi with:"
echo " sshpass -p '$PI_PASS' ssh $PI_USER@$PI_HOST \\"
echo " 'umask 077 && printf \"%s\\n\" \"sk-PASTE-YOUR-KEY\" > ~/.openai_key'"
exit 1
fi
echo " OK."
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/04a-realtime-oneshot-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-4a/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-4a/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-4a && chmod +x Probe && ./Probe'
@@ -0,0 +1,758 @@
# Test 3 — Full-cycle C# probe Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a single-binary C# probe at `tests/03-full-cycle-cs/` that runs the full assistant cycle on the Pi: idle → "alexa" → beep → record 5 s → play back → idle, with detector gating during non-idle states and a `WakewordModel.Reset()` on every IDLE re-entry.
**Architecture:** One persistent PortAudio input stream feeds 80 ms frames into a `BlockingCollection`; a single consumer thread runs a four-state machine (IDLE / BEEPING / RECORDING / PLAYBACK). Beep and playback are fire-and-forget output streams using Test 2's `Start()`-before-`Task.Run` + 2 s timeout lifetime pattern. `WakewordModel` is copied verbatim from Test 2 with a new `Reset()` method called on PLAYBACK→IDLE.
**Tech Stack:** .NET 9 (self-contained linux-arm64), PortAudioSharp2 1.0.6, Microsoft.ML.OnnxRuntime 1.26.0. Deploy via `dotnet publish` + `scp` + `ssh -t` to Pi at `pi@192.168.50.115`.
**Spec:** `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`. Read it before starting.
**Branch:** `fresh-start` (already current; do NOT create a new branch).
---
### Task 1: Scaffold `tests/03-full-cycle-cs/` project skeleton
**Files:**
- Create: `tests/03-full-cycle-cs/Probe.csproj`
- Create: `tests/03-full-cycle-cs/models/` (directory; populated in Task 2)
- [ ] **Step 1: Create the project directory**
```bash
mkdir -p tests/03-full-cycle-cs/models
```
- [ ] **Step 2: Write `Probe.csproj`**
Create `tests/03-full-cycle-cs/Probe.csproj` with this exact content (identical to `tests/02-wakeword-cs/Probe.csproj` except `RootNamespace`):
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>FullCycleProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.26.0" />
</ItemGroup>
<ItemGroup>
<Content Include="models/*.onnx">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project>
```
- [ ] **Step 3: Commit the scaffold**
```bash
git add tests/03-full-cycle-cs/Probe.csproj
git commit -m "Test 3 scaffold: csproj for tests/03-full-cycle-cs"
```
---
### Task 2: Copy models and `WakewordModel.cs` from Test 2 (with namespace rename and `Reset()`)
**Files:**
- Create: `tests/03-full-cycle-cs/WakewordModel.cs` (copied from Test 2, namespace renamed, `Reset()` added)
- Create: `tests/03-full-cycle-cs/models/melspectrogram.onnx`
- Create: `tests/03-full-cycle-cs/models/embedding_model.onnx`
- Create: `tests/03-full-cycle-cs/models/alexa.onnx`
- [ ] **Step 1: Copy the three ONNX model files**
```bash
cp tests/02-wakeword-cs/models/melspectrogram.onnx tests/03-full-cycle-cs/models/
cp tests/02-wakeword-cs/models/embedding_model.onnx tests/03-full-cycle-cs/models/
cp tests/02-wakeword-cs/models/alexa.onnx tests/03-full-cycle-cs/models/
```
- [ ] **Step 2: Copy `WakewordModel.cs`**
```bash
cp tests/02-wakeword-cs/WakewordModel.cs tests/03-full-cycle-cs/WakewordModel.cs
```
- [ ] **Step 3: Rename the namespace in the copy**
Edit `tests/03-full-cycle-cs/WakewordModel.cs`: change line 4 from `namespace WakewordProbe;` to `namespace FullCycleProbe;`. Leave everything else unchanged.
- [ ] **Step 4: Add `Reset()` method to the copy**
Edit `tests/03-full-cycle-cs/WakewordModel.cs`. Insert the following method immediately before the `public void Dispose()` method near the bottom of the file:
```csharp
public void Reset()
{
_rawRingFill = 0;
_melRing.Clear();
_embRing.Clear();
_framesSeen = 0;
}
```
(Note the trailing blank line. The result should be a method block sitting between `return clsOut[0, 0]; }` (end of `Predict`) and `public void Dispose()`.)
- [ ] **Step 5: Verify the project builds**
Run from repo root:
```bash
dotnet build tests/03-full-cycle-cs -c Release
```
Expected: `Build succeeded` with 0 errors, 0 warnings. (At this point `Program.cs` doesn't exist yet; csproj is `<OutputType>Exe</OutputType>` but the build will produce a no-entry-point warning or error. If it errors with "CS5001: Program does not contain a static 'Main' method", that's expected at this stage — proceed to Task 3 which adds `Program.cs`.)
If the build errors for any reason other than the missing entry point, fix before continuing.
- [ ] **Step 6: Commit**
```bash
git add tests/03-full-cycle-cs/WakewordModel.cs tests/03-full-cycle-cs/models/
git commit -m "Test 3: copy WakewordModel + models from Test 2, add Reset()"
```
---
### Task 3: Write `Program.cs` with the full state machine
**Files:**
- Create: `tests/03-full-cycle-cs/Program.cs`
The whole file is shown below. It lifts from `tests/02-wakeword-cs/Program.cs` (env-var setup, OrtEnv init, FindUsbDevice, input stream + queue, FireAndForgetBeep, Libc, MakeBeep) and adds:
- `volatile State _state` enum-as-state-machine in the consumer loop.
- `FireAndForgetPlayback` helper (same shape as `FireAndForgetBeep` but reads from `recordBuf` and signals via `_playbackDoneFlag` instead of a `ManualResetEventSlim`).
- `model.Reset()` call on PLAYBACK→IDLE transition.
- [ ] **Step 1: Write `Program.cs` in full**
Create `tests/03-full-cycle-cs/Program.cs` with exactly this content:
```csharp
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Runtime.InteropServices;
using FullCycleProbe;
using Microsoft.ML.OnnxRuntime;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
const int SampleRate = WakewordModel.SampleRate; // 16 000
const int Channels = 1;
const uint BlockFrames = WakewordModel.FrameSamples; // 1280 (80 ms)
const float Threshold = 0.5f;
const double CooldownSeconds = 1.0;
const double BeepHz = 880.0;
const double BeepSeconds = 0.2;
const int BeepingFrames = 3; // 3 * 80 ms = 240 ms (covers 200 ms beep + 40 ms drain/reverb margin)
const int RecordingFrames = 63; // 63 * 80 ms = 5.04 s (rounded up from 5 s to avoid partial-frame handling)
const int RecordBufSamples = RecordingFrames * (int)BlockFrames; // 80640
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio / onnxruntime (which use getenv()) see the values.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
Environment.SetEnvironmentVariable("ORT_LOGGING_LEVEL", "3");
Libc.setenv("ORT_LOGGING_LEVEL", "3", 1);
// ORT 1.26.0: the env var does NOT suppress early GPU-discovery warnings from device_discovery.cc.
// CreateInstanceWithOptions sets the log level at OrtEnv creation time, before EP discovery runs.
var ortEnvOpts = new EnvironmentCreationOptions { logId = "FullCycleProbe", logLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR };
OrtEnv.CreateInstanceWithOptions(ref ortEnvOpts);
PortAudio.Initialize();
WakewordModel? model = null;
try
{
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
Console.WriteLine("Loading WakewordModel...");
var t0 = DateTime.UtcNow;
string modelsDir = Path.Combine(AppContext.BaseDirectory, "models");
model = new WakewordModel(
Path.Combine(modelsDir, "melspectrogram.onnx"),
Path.Combine(modelsDir, "embedding_model.onnx"),
Path.Combine(modelsDir, "alexa.onnx"));
Console.WriteLine($"Loaded in {(DateTime.UtcNow - t0).TotalSeconds:0.00}s.");
var queue = new BlockingCollection<short[]>(boundedCapacity: 16);
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var inParams = new StreamParameters
{
device = device,
channelCount = Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (frameCount != BlockFrames)
{
Console.Error.WriteLine($"[status] unexpected callback frameCount={frameCount} (want {BlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[BlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
if (!queue.TryAdd(frame, 0))
Console.Error.WriteLine("[status] consumer behind, dropping frame");
return StreamCallbackResult.Continue;
};
using var stream = new Stream(
inParams, null, SampleRate, BlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
stream.Start();
short[] beepBuffer = MakeBeep(BeepHz, BeepSeconds, SampleRate);
Console.WriteLine("Listening. Say 'alexa'. Ctrl-C to exit.");
var sw = Stopwatch.StartNew();
TimeSpan lastTrigger = TimeSpan.FromSeconds(-CooldownSeconds);
// State machine (single-writer: this consumer thread).
State state = State.Idle;
int beepFrames = 0;
int recordOffset = 0;
short[] recordBuf = null!; // assigned on BEEPING→RECORDING transition
var playbackDone = new PlaybackDoneFlag();
try
{
while (!cts.IsCancellationRequested)
{
short[] frame = queue.Take(cts.Token);
switch (state)
{
case State.Idle:
{
float score = model.Predict(frame);
var now = sw.Elapsed;
if (score >= Threshold && (now - lastTrigger).TotalSeconds >= CooldownSeconds)
{
Console.WriteLine($"DETECTED alexa score={score:0.000} t={now.TotalSeconds:0.0}s");
lastTrigger = now;
try
{
FireAndForgetBeep(device, beepBuffer);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] beep stream failed to start: {ex.Message}");
// Stay in IDLE — no cycle to abort yet.
break;
}
Console.WriteLine("[state] BEEPING");
state = State.Beeping;
beepFrames = 0;
}
break;
}
case State.Beeping:
{
beepFrames++;
if (beepFrames >= BeepingFrames)
{
Console.WriteLine("[state] RECORDING");
state = State.Recording;
recordBuf = new short[RecordBufSamples];
recordOffset = 0;
}
break;
}
case State.Recording:
{
unsafe
{
fixed (short* src = &frame[0])
fixed (short* dst = &recordBuf[recordOffset])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
recordOffset += (int)BlockFrames;
if (recordOffset >= RecordBufSamples)
{
try
{
FireAndForgetPlayback(device, recordBuf, playbackDone);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] playback stream failed to start: {ex.Message}");
// Drop the recording, reset model, return to IDLE.
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
break;
}
Console.WriteLine("[state] PLAYBACK");
state = State.Playback;
}
break;
}
case State.Playback:
{
// Discard input during playback (gate the detector).
if (playbackDone.Consume())
{
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
}
break;
}
}
}
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
stream.Stop();
Console.WriteLine("\nbye");
}
finally
{
model?.Dispose();
PortAudio.Terminate();
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static void FireAndForgetBeep(int device, short[] beepBuffer)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = beepBuffer.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &beepBuffer[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= beepBuffer.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
Task.Run(() =>
{
// 2 s timeout so a failed callback (which PortAudio silently swallows
// and would leave 'done' unset) eventually releases the stream + handle
// instead of leaking them for the process lifetime.
done.Wait(TimeSpan.FromSeconds(2));
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
});
}
static void FireAndForgetPlayback(int device, short[] recordBuf, PlaybackDoneFlag doneFlag)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = recordBuf.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &recordBuf[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= recordBuf.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
Task.Run(() =>
{
// Same 2 s safety timeout as the beep cleanup.
done.Wait(TimeSpan.FromSeconds(2));
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
doneFlag.Set(); // tell the consumer it's safe to flip PLAYBACK→IDLE
});
}
enum State { Idle, Beeping, Recording, Playback }
// Volatile flag set by the playback cleanup task and consumed by the consumer thread.
// Wrapped in a tiny class so we can pass it by reference into the helper.
sealed class PlaybackDoneFlag
{
private volatile bool _set;
public void Set() => _set = true;
public bool Consume()
{
if (!_set) return false;
_set = false;
return true;
}
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}
```
- [ ] **Step 2: Verify the project builds cleanly**
```bash
dotnet build tests/03-full-cycle-cs -c Release
```
Expected: `Build succeeded` with 0 errors, 0 warnings.
If `<AllowUnsafeBlocks>` is missing from the csproj, the build will fail with `CS0227: Unsafe code may only appear if compiling with /unsafe`. Confirm Task 1's csproj has `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`.
If you see `CS0246: The type or namespace 'Stream' could not be found` or a clash with `System.IO.Stream`, confirm the `using Stream = PortAudioSharp.Stream;` line is at the top of `Program.cs`.
- [ ] **Step 3: Verify `dotnet publish` produces a runnable binary**
```bash
dotnet publish tests/03-full-cycle-cs -c Release -r linux-arm64 --self-contained -o /tmp/probe-cs-3-out
ls /tmp/probe-cs-3-out/Probe
ls /tmp/probe-cs-3-out/models/
```
Expected:
- `Probe` (executable) exists.
- `models/` directory contains `melspectrogram.onnx`, `embedding_model.onnx`, `alexa.onnx`.
- [ ] **Step 4: Commit**
```bash
git add tests/03-full-cycle-cs/Program.cs
git commit -m "Test 3: full-cycle state machine (IDLE/BEEPING/RECORDING/PLAYBACK)"
```
---
### Task 4: Write `bin/probe-cs-3` deploy script
**Files:**
- Create: `bin/probe-cs-3`
- [ ] **Step 1: Write the deploy script**
Create `bin/probe-cs-3` with exactly this content (it's `bin/probe-cs-2` with three string swaps: `02-wakeword-cs``03-full-cycle-cs`, `probe-cs-2``probe-cs-3`, `/tmp/probe-cs-2-out``/tmp/probe-cs-3-out`):
```bash
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-3-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/03-full-cycle-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-3/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-3 && mkdir ~/probe-cs-3'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-3/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-3 && chmod +x Probe && ./Probe'
```
- [ ] **Step 2: Make it executable**
```bash
chmod +x bin/probe-cs-3
```
- [ ] **Step 3: Commit**
```bash
git add bin/probe-cs-3
git commit -m "Test 3: bin/probe-cs-3 deploy script"
```
---
### Task 5: Hardware verification on the Pi
This task is the actual pass/fail gate. No automated tests; verification is the human ear + console output + `top` reading on the Pi. If anything in this task fails, do NOT mark the implementation complete — investigate and either fix in code or update the spec to reflect what actually works.
**Files:** (none — this is a runtime verification task)
- [ ] **Step 1: Confirm the Pi is reachable**
Run from workstation:
```bash
ping -c 2 192.168.50.115
```
Expected: 2/2 replies. If unreachable, stop and ask the user.
- [ ] **Step 2: Deploy + run**
Run from workstation:
```bash
./bin/probe-cs-3
```
Expected console output (abridged):
```
>> dotnet publish (linux-arm64, self-contained)
... (build output) ...
>> scp to pi@192.168.50.115:~/probe-cs-3/ (wipe-and-replace)
>> ssh + run on Pi (Ctrl-C from this terminal stops the probe)
[wakeword-model] melspectrogram.onnx
input ...
[wakeword-model] embedding_model.onnx ...
[wakeword-model] alexa.onnx ...
Loaded in 0.6s.
Using device <N> ('USB Speaker Phone...')
Listening. Say 'alexa'. Ctrl-C to exit.
```
If the publish step errors with `setenv: command not found` or similar libc-related runtime crash, confirm Task 3's `Libc.setenv` `[DllImport]` is present and the binary was self-contained-published.
If the Pi run errors with `Cannot open shared library libonnxruntime.so` or `libportaudio.so`, the self-contained publish didn't bundle the native libs — confirm `<SelfContained>true</SelfContained>` and `<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>` in Task 1's csproj.
- [ ] **Step 3: Run 3 full cycles back-to-back**
Say "alexa" clearly at normal volume from ~1 m. Observe:
1. Beep plays within ~100 ms of saying "alexa".
2. Console prints `DETECTED alexa score=…`, then `[state] BEEPING`, then `[state] RECORDING`.
3. Say something distinctive (e.g. "hello world, this is cycle one") within the 5 s recording window.
4. Console prints `[state] PLAYBACK`.
5. Speaker plays back what you said. It should be recognisable.
6. Console prints `[state] IDLE`.
7. Pause ~1.5 s, then say "alexa" again. Verify the cycle starts over.
Repeat for cycle 2 and cycle 3. Record on paper (or transcript) which cycles succeeded.
Pass criterion 1: 3 full cycles back-to-back; each playback recognisable as what was spoken.
- [ ] **Step 4: Measure idle CPU on the Pi**
In a second workstation terminal:
```bash
sshpass -p 'assistant' ssh pi@192.168.50.115 'top -b -n 3 -d 1 -p $(pgrep -f Probe)'
```
Expected: `Probe` process appears with `%CPU` < 50 across the three samples (Test 2 measured 1731%, Test 3 should be in the same range during IDLE).
Pass criterion 4: CPU < 50% of one core during IDLE.
- [ ] **Step 5: Test post-playback re-detection latency**
Speak "alexa" immediately after each `[state] IDLE` log line. Note how soon detection fires.
Pass criterion 2: Next "alexa" after playback detected within ~2 s of playback ending. (Below ~1.3 s is impossible due to WarmupFrames; above ~2 s indicates a regression.)
- [ ] **Step 6: Scan for warnings**
Re-read the console output from Step 3. Confirm there are **no** lines of the form:
- `[status] input overflow`
- `[status] consumer behind, dropping frame`
- `[status] unexpected callback frameCount=...`
- `[error] beep stream failed to start: ...`
- `[error] playback stream failed to start: ...`
A single occurrence under abnormal load may be tolerable; sustained occurrences during normal cycles fail criterion 5.
- [ ] **Step 7: Test Ctrl-C clean exit**
Press Ctrl-C in the workstation terminal that's running the probe.
Expected:
```
^C
bye
$
```
Pass criterion 7: process exits 0, no stack trace, terminal returns to prompt. Verify with `echo $?``0`.
- [ ] **Step 8: Wakeword detection rate during IDLE**
Restart the probe (`./bin/probe-cs-3`). Wait until `Listening. Say 'alexa'. Ctrl-C to exit.` appears. Say "alexa" 10 times with a ~2 s gap between each. Count how many times `DETECTED alexa score=…` appears.
Pass criterion 3: ≥ 8/10. (Test 2 demonstrated this rate; Test 3 should match.)
- [ ] **Step 9: Update `findings.md` with the outcome**
Add a new section `## Test 3 full-cycle outcome (2026-06-12)` at the bottom of `findings.md` summarising:
- Pass / fail per criterion.
- Any new gotchas discovered during implementation or hardware verification (especially any that contradicted the spec's predictions).
- CPU measurement (peak + typical), detection latency post-playback, observed pass rate.
- Reference paths (spec + plan + code + deploy script).
Use the existing Test 1 and Test 2 outcome sections in `findings.md` as the template.
- [ ] **Step 10: Commit the findings update**
```bash
git add findings.md
git commit -m "Findings: Test 3 full-cycle probe outcome"
```
- [ ] **Step 11: Final status report**
Report to the user:
- Pass/fail per criterion (1 through 7 from spec).
- Any criteria that failed and what was discovered.
- A one-line summary of whether Test 3 unblocks the main Pi-client implementation.
---
## Self-review notes
- **Spec coverage:** every section of the spec maps to a task — Architecture/State machine/Components → Tasks 14. Why-Reset-matters → Task 2's `Reset()` + Task 3's PLAYBACK→IDLE branch. Error handling → Task 3's try/catch on the FireAndForget calls + `[status]` / `[error]` log lines. Gotchas-carried-forward → preserved verbatim in Task 3's code. Verification → Task 5.
- **Placeholder scan:** no TBD/TODO; every code step shows the actual code; every command shows expected output.
- **Type consistency:** `State` enum (`Idle`, `Beeping`, `Recording`, `Playback`) used consistently; `PlaybackDoneFlag.Set()` / `.Consume()` used consistently between `FireAndForgetPlayback` and the consumer's PLAYBACK branch; `WakewordModel.Reset()` called in two places (Recording-failure recovery and Playback→Idle), signature is parameterless in both.
@@ -0,0 +1,897 @@
# Test 4a — OpenAI Realtime API one-shot voice round-trip (C#) Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a single-binary throwaway C# probe at `tests/04a-realtime-oneshot-cs/` that has one round-trip spoken conversation with the OpenAI Realtime API: boot → beep → user speaks one sentence → server VAD ends the turn → assistant audio reply plays back through the USB Speaker Phone → exit 0.
**Architecture:** One persistent `ClientWebSocket` to `wss://api.openai.com/v1/realtime?model=gpt-realtime`. Two `Task.Run` loops: ReceiveLoop parses server events with `JsonDocument` and writes decoded audio chunks to a `Channel<short[]>`; UpstreamSendLoop drains a separate `Channel<short[]>` of mic frames and sends them as `input_audio_buffer.append`. PortAudio input stream (24 kHz mono Int16, 1920 frames/cb) writes to the upstream channel via callback; output stream (24 kHz, 1024 frames/cb) drains the downstream channel into the speaker, zero-filling underruns. Completion (`response.done` received AND downstream channel drained AND working chunk empty) is signaled via `ManualResetEventSlim` to the main thread, which closes the WS and exits.
**Tech Stack:** .NET 9 (self-contained linux-arm64), PortAudioSharp2 1.0.6, `System.Net.WebSockets.ClientWebSocket`, `System.Text.Json`, `System.Threading.Channels`. Deploy via `dotnet publish` + `scp` + `ssh -t` to Pi at `pi@192.168.50.115`. No onnxruntime — no wakeword in 4a.
**Spec:** `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`. Read it before starting.
**Branch:** `fresh-start` (already current; do NOT create a new branch).
---
### Task 1: Scaffold `tests/04a-realtime-oneshot-cs/` project skeleton
**Files:**
- Create: `tests/04a-realtime-oneshot-cs/Probe.csproj`
- [ ] **Step 1: Create the project directory**
```bash
mkdir -p tests/04a-realtime-oneshot-cs
```
- [ ] **Step 2: Write `Probe.csproj`**
Create `tests/04a-realtime-oneshot-cs/Probe.csproj` with this exact content. Note the differences from Test 3's csproj: `RootNamespace=RealtimeOneShotProbe`, no `Microsoft.ML.OnnxRuntime` reference (no wakeword in 4a), no `models/*.onnx` content block.
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>RealtimeOneShotProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
</ItemGroup>
</Project>
```
`ClientWebSocket` (from `System.Net.WebSockets`), `JsonDocument` / `JsonSerializer` (from `System.Text.Json`), and `Channel<T>` (from `System.Threading.Channels`) all come from the BCL — no PackageReference needed.
- [ ] **Step 3: Commit the scaffold**
```bash
git add tests/04a-realtime-oneshot-cs/Probe.csproj
git commit -m "Test 4a scaffold: csproj for tests/04a-realtime-oneshot-cs"
```
---
### Task 2: Write `Program.cs` in full
**Files:**
- Create: `tests/04a-realtime-oneshot-cs/Program.cs`
The whole file is shown below. It is self-contained — no helper files, no copied modules. The structure follows Test 3's `Program.cs` shape (top-level statements; helpers, classes, and `static class Libc` at the bottom of the file) with these notable differences:
- No onnxruntime, no `WakewordModel`, no models directory.
- A `ClientWebSocket` + two background `Task.Run` loops (ReceiveLoop, UpstreamSendLoop) replace the consumer-thread state machine.
- The input stream callback's job is to push mic frames to a `Channel<short[]>` (not to call `Predict`).
- The output stream callback drains a different `Channel<short[]>`, zero-fills underruns, and signals completion via a `ManualResetEventSlim` (replacing Test 3's per-cycle `PlaybackDoneFlag`).
- A `SharedState` class holds the three `volatile bool` flags (`MicArmed`, `StopSending`, `NoMoreDeltas`) plus the output callback's per-call drain state (`CurrentChunk`, `ChunkOffset`).
- `LoadApiKey()` reads `~/.openai_key` with mode-600 validation.
- [ ] **Step 1: Write `Program.cs` in full**
Create `tests/04a-realtime-oneshot-cs/Program.cs` with exactly this content:
```csharp
using System.Net.WebSockets;
using System.Runtime.InteropServices;
using System.Text;
using System.Text.Json;
using System.Threading.Channels;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio (which uses getenv()) sees the value.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
string apiKey = LoadApiKey();
PortAudio.Initialize();
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var upstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.UpstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = true,
SingleReader = true,
});
var downstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.DownstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = false,
SingleReader = true,
});
var doneFlag = new ManualResetEventSlim(false);
var state = new SharedState();
var ws = new ClientWebSocket();
ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
ws.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");
Stream? inputStream = null;
Stream? outputStream = null;
Task? receiveTask = null;
Task? sendTask = null;
try
{
var uri = new Uri($"wss://api.openai.com/v1/realtime?model={C.ModelName}");
Console.WriteLine($"[ws] connecting to {uri}");
try
{
await ws.ConnectAsync(uri, cts.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] WS connect failed: {ex.Message}");
if (ex.InnerException is HttpRequestException httpEx)
Console.Error.WriteLine($"[error] inner: HTTP {(int?)httpEx.StatusCode}");
Environment.ExitCode = 3;
return;
}
Console.WriteLine("[ws] connected");
receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel, cts, state));
await SendSessionUpdate(ws, cts.Token);
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
inputStream = OpenInputStream(device, upstreamChannel, state);
try { inputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] input stream start failed: {ex.Message}");
Environment.ExitCode = 5;
return;
}
outputStream = OpenOutputStream(device, downstreamChannel, state, doneFlag);
try { outputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] output stream start failed: {ex.Message}");
Environment.ExitCode = 6;
return;
}
sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel, cts, state));
// Startup beep — queued onto downstreamChannel before the receive loop produces any audio.
short[] beep = MakeBeep(C.BeepHz, C.BeepSeconds, C.SampleRate);
Console.WriteLine("[beep]");
await downstreamChannel.Writer.WriteAsync(beep, cts.Token);
// Wait for beep to drain (200 ms beep + 100 ms margin) so it doesn't get captured
// by the mic and shipped to OpenAI.
Thread.Sleep((int)(C.BeepSeconds * 1000) + C.BeepDrainMs);
state.MicArmed = true;
Console.WriteLine("Speak now.");
try
{
doneFlag.Wait(cts.Token);
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
Console.WriteLine("[ws] closing");
}
finally
{
// Cancel any background work first so receive/send loops exit cleanly.
cts.Cancel();
if (ws.State == WebSocketState.Open || ws.State == WebSocketState.CloseReceived)
{
try
{
using var closeTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(2));
await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", closeTimeout.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[ws] close failed: {ex.Message}");
}
}
try { inputStream?.Stop(); } catch { }
try { outputStream?.Stop(); } catch { }
try { inputStream?.Dispose(); } catch { }
try { outputStream?.Dispose(); } catch { }
try { ws.Dispose(); } catch { }
try { PortAudio.Terminate(); } catch { }
Console.WriteLine("bye");
}
static string LoadApiKey()
{
string home = Environment.GetEnvironmentVariable("HOME") ?? "/root";
string path = Path.Combine(home, ".openai_key");
if (!File.Exists(path))
{
Console.Error.WriteLine($"[error] API key file not found at {path}");
Environment.Exit(2);
}
// Non-fatal mode-600 check.
try
{
var mode = File.GetUnixFileMode(path);
var leaky = mode & (UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute
| UnixFileMode.OtherRead | UnixFileMode.OtherWrite | UnixFileMode.OtherExecute);
if (leaky != UnixFileMode.None)
Console.Error.WriteLine($"[warn] {path} mode includes group/other perms ({mode}); expected 600");
}
catch (PlatformNotSupportedException) { /* not Unix */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[warn] could not check {path} mode: {ex.Message}");
}
string key = File.ReadAllText(path).Trim();
if (string.IsNullOrEmpty(key))
{
Console.Error.WriteLine($"[error] API key file empty");
Environment.Exit(2);
}
if (!key.StartsWith("sk-"))
{
Console.Error.WriteLine($"[error] API key doesn't start with sk-");
Environment.Exit(2);
}
return key;
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static async Task SendSessionUpdate(ClientWebSocket ws, CancellationToken ct)
{
var msg = new
{
type = "session.update",
session = new
{
modalities = new[] { "audio", "text" },
instructions = C.Instructions,
voice = C.Voice,
input_audio_format = "pcm16",
output_audio_format = "pcm16",
turn_detection = new
{
type = "server_vad",
threshold = 0.5,
prefix_padding_ms = 300,
silence_duration_ms = 500,
create_response = true,
},
},
};
byte[] bytes = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(bytes), WebSocketMessageType.Text, endOfMessage: true, ct);
}
static Stream OpenInputStream(int device, Channel<short[]> upstream, SharedState state)
{
var inParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (!state.MicArmed || state.StopSending)
return StreamCallbackResult.Continue;
if (frameCount != C.MicBlockFrames)
{
Console.Error.WriteLine($"[status] unexpected input frameCount={frameCount} (want {C.MicBlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[C.MicBlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, C.MicBlockFrames * sizeof(short), C.MicBlockFrames * sizeof(short));
}
if (!upstream.Writer.TryWrite(frame))
Console.Error.WriteLine("[status] upstream behind, dropping mic frame");
return StreamCallbackResult.Continue;
};
return new Stream(inParams, null, C.SampleRate, C.MicBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static Stream OpenOutputStream(int device, Channel<short[]> downstream, SharedState state, ManualResetEventSlim doneFlag)
{
var outParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int fill = 0;
while (fill < frameCount)
{
// Pull next chunk if we have no working one.
if (state.CurrentChunk == null)
{
if (!downstream.Reader.TryRead(out var nextChunk))
break; // ring empty
state.CurrentChunk = nextChunk;
state.ChunkOffset = 0;
}
// Local non-null alias so the compiler doesn't warn on dereference inside `fixed`.
short[] chunk = state.CurrentChunk!;
int slack = (int)frameCount - fill;
int remaining = chunk.Length - state.ChunkOffset;
int take = Math.Min(remaining, slack);
unsafe
{
short* dst = (short*)output.ToPointer();
fixed (short* src = &chunk[state.ChunkOffset])
Buffer.MemoryCopy(src, dst + fill, take * sizeof(short), take * sizeof(short));
}
state.ChunkOffset += take;
fill += take;
if (state.ChunkOffset >= chunk.Length)
state.CurrentChunk = null;
}
unsafe
{
short* dst = (short*)output.ToPointer();
for (int i = fill; i < (int)frameCount; i++) dst[i] = 0;
}
// Completion = working chunk drained AND channel empty AND writer completed.
if (state.CurrentChunk == null
&& downstream.Reader.Count == 0
&& downstream.Reader.Completion.IsCompleted)
{
doneFlag.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
return new Stream(null, outParams, C.SampleRate, C.OutBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static async Task UpstreamSendLoop(ClientWebSocket ws, Channel<short[]> upstream, CancellationTokenSource cts, SharedState state)
{
try
{
await foreach (var frame in upstream.Reader.ReadAllAsync(cts.Token))
{
if (state.StopSending) continue;
ReadOnlySpan<byte> bytes = MemoryMarshal.AsBytes(frame.AsSpan());
string base64 = Convert.ToBase64String(bytes);
var msg = new { type = "input_audio_buffer.append", audio = base64 };
byte[] json = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(json), WebSocketMessageType.Text, endOfMessage: true, cts.Token);
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] upstream send: {ex.Message}");
cts.Cancel();
}
}
static async Task ReceiveLoop(ClientWebSocket ws, Channel<short[]> downstream, CancellationTokenSource cts, SharedState state)
{
var transcript = new StringBuilder();
var buffer = new byte[4096];
bool firstDeltaSeen = false;
DateTime speechStoppedTime = DateTime.MinValue;
using var msgBuffer = new MemoryStream();
try
{
while (!cts.IsCancellationRequested)
{
msgBuffer.SetLength(0);
WebSocketReceiveResult result;
do
{
result = await ws.ReceiveAsync(new ArraySegment<byte>(buffer), cts.Token);
if (result.MessageType == WebSocketMessageType.Close) return;
msgBuffer.Write(buffer, 0, result.Count);
} while (!result.EndOfMessage);
msgBuffer.Position = 0;
using var doc = JsonDocument.Parse(msgBuffer);
string type = doc.RootElement.GetProperty("type").GetString() ?? "";
switch (type)
{
case "session.created":
{
string? sid = doc.RootElement.TryGetProperty("session", out var sess)
&& sess.TryGetProperty("id", out var idEl) ? idEl.GetString() : null;
Console.WriteLine($"[ws] session.created {sid}");
break;
}
case "session.updated":
Console.WriteLine("[ws] session.updated — session pinned");
break;
case "input_audio_buffer.speech_started":
Console.WriteLine("[vad] speech started");
break;
case "input_audio_buffer.speech_stopped":
Console.WriteLine("[vad] speech stopped");
speechStoppedTime = DateTime.UtcNow;
state.StopSending = true;
break;
case "input_audio_buffer.committed":
Console.WriteLine("[ws] input_audio_buffer.committed");
break;
case "response.created":
Console.WriteLine("[ws] response.created");
break;
case "response.audio_transcript.delta":
if (doc.RootElement.TryGetProperty("delta", out var tEl))
transcript.Append(tEl.GetString());
break;
case "response.audio.delta":
{
string b64 = doc.RootElement.GetProperty("delta").GetString() ?? "";
byte[] audioBytes = Convert.FromBase64String(b64);
short[] samples = new short[audioBytes.Length / 2];
Buffer.BlockCopy(audioBytes, 0, samples, 0, audioBytes.Length);
if (!firstDeltaSeen)
{
firstDeltaSeen = true;
double elapsed = speechStoppedTime == DateTime.MinValue
? -1
: (DateTime.UtcNow - speechStoppedTime).TotalSeconds;
Console.WriteLine($"[audio] first delta received (t = {elapsed:0.00} s since speech_stopped)");
}
await downstream.Writer.WriteAsync(samples, cts.Token);
break;
}
case "response.audio.done":
Console.WriteLine("[ws] response.audio.done");
break;
case "response.done":
state.NoMoreDeltas = true;
downstream.Writer.Complete();
Console.WriteLine("[ws] response.done");
Console.WriteLine($"[transcript] {transcript}");
break;
case "error":
Console.Error.WriteLine($"[error] WS error event: {doc.RootElement.GetRawText()}");
cts.Cancel();
break;
default:
Console.Error.WriteLine($"[ws] ignored type={type}");
break;
}
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] receive loop: {ex.Message}");
cts.Cancel();
}
}
sealed class SharedState
{
public volatile bool MicArmed;
public volatile bool StopSending;
public volatile bool NoMoreDeltas;
// Output-callback-thread-local; do NOT touch from anywhere else.
public short[]? CurrentChunk;
public int ChunkOffset;
}
static class C
{
public const int SampleRate = 24_000;
public const int Channels = 1;
public const uint MicBlockFrames = 1920; // 40 ms @ 24 kHz
public const uint OutBlockFrames = 1024; // ~43 ms @ 24 kHz
public const string ModelName = "gpt-realtime";
public const string Voice = "alloy";
public const string Instructions = "Reply in one short sentence.";
public const double BeepHz = 880.0;
public const double BeepSeconds = 0.2;
public const int BeepDrainMs = 100;
public const int UpstreamChannelCapacity = 64;
public const int DownstreamChannelCapacity = 64;
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}
```
- [ ] **Step 2: Verify the project builds cleanly**
```bash
dotnet build tests/04a-realtime-oneshot-cs -c Release
```
Expected: `Build succeeded` with 0 errors. A handful of nullable-reference warnings on `state.CurrentChunk` access inside the `unsafe` block is fine if it appears; the runtime checks above ensure non-null.
If `<AllowUnsafeBlocks>` is missing from the csproj, the build will fail with `CS0227: Unsafe code may only appear if compiling with /unsafe`. Confirm Task 1's csproj has `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`.
If you see `CS0246: The type or namespace 'Stream' could not be found` or a clash with `System.IO.Stream`, confirm the `using Stream = PortAudioSharp.Stream;` line is at the top of `Program.cs`.
If you see `error CS8803: Top-level statements must precede namespace and type declarations` or `CS8805: Program using top-level statements must be an entry point`, the file ordering is wrong. The exact required order in `Program.cs` is:
1. `using` directives
2. Top-level statements (the imperative body — `Environment.SetEnvironmentVariable`, `apiKey = LoadApiKey()`, `PortAudio.Initialize`, the `try {} finally {}` block)
3. Top-level static method declarations (`LoadApiKey`, `FindUsbDevice`, `MakeBeep`, `SendSessionUpdate`, `OpenInputStream`, `OpenOutputStream`, `UpstreamSendLoop`, `ReceiveLoop`)
4. Top-level type declarations (`sealed class SharedState`, `static class C`, `static class Libc`) — NOT wrapped in a `namespace` block.
The file above follows this order; do not reorder.
- [ ] **Step 3: Verify `dotnet publish` produces a runnable binary**
```bash
dotnet publish tests/04a-realtime-oneshot-cs -c Release -r linux-arm64 --self-contained -o /tmp/probe-cs-4a-out
ls /tmp/probe-cs-4a-out/Probe
```
Expected:
- `Probe` (executable) exists.
- No `models/` directory needed.
- [ ] **Step 4: Commit**
```bash
git add tests/04a-realtime-oneshot-cs/Program.cs
git commit -m "Test 4a: WS plumbing + audio streams + server VAD round-trip"
```
---
### Task 3: Write `bin/probe-cs-4a` deploy script with API key precondition
**Files:**
- Create: `bin/probe-cs-4a`
This is `bin/probe-cs-3` with three string swaps (`03-full-cycle-cs``04a-realtime-oneshot-cs`, `probe-cs-3``probe-cs-4a`, `/tmp/probe-cs-3-out``/tmp/probe-cs-4a-out`) plus ONE new precondition step: check that `~/.openai_key` exists on the Pi before publishing. This catches the silent "key missing" failure mode at the deploy step instead of 30 s into publish + scp.
- [ ] **Step 1: Write the deploy script**
Create `bin/probe-cs-4a` with exactly this content:
```bash
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-4a-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> precondition: ~/.openai_key present on Pi"
if ! sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'test -f ~/.openai_key && test -r ~/.openai_key'; then
echo "ERROR: ~/.openai_key missing or unreadable on Pi."
echo "Create it on the Pi with:"
echo " sshpass -p '$PI_PASS' ssh $PI_USER@$PI_HOST \\"
echo " 'umask 077 && printf \"%s\\n\" \"sk-PASTE-YOUR-KEY\" > ~/.openai_key'"
exit 1
fi
echo " OK."
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/04a-realtime-oneshot-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-4a/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-4a/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-4a && chmod +x Probe && ./Probe'
```
- [ ] **Step 2: Make it executable**
```bash
chmod +x bin/probe-cs-4a
```
- [ ] **Step 3: Commit**
```bash
git add bin/probe-cs-4a
git commit -m "Test 4a: bin/probe-cs-4a deploy script with ~/.openai_key precondition"
```
---
### Task 4: One-time Pi setup — place `~/.openai_key` on the Pi
**Files:** (none — this is a one-time remote setup task)
This task only runs once per Pi. If the file already exists from a previous deploy or another test, skip steps 12 and verify with step 3.
- [ ] **Step 1: Ask the user for an API key**
Before running this task, the user must have an OpenAI API key with Realtime API access (`sk-…`). If the user hasn't given it to you yet, ask:
> "I need your OpenAI API key (`sk-…`) to write it to `~/.openai_key` on the Pi. Paste it and I'll redact it from the shell history immediately. Do not paste it into any other context."
The key the user provides MUST be treated as secret and MUST NOT appear in commit messages, file contents anywhere in the repo, console echoes, or logs. The only place it goes is the file on the Pi.
- [ ] **Step 2: Write the key to the Pi**
Replace `sk-PASTE-YOUR-KEY-HERE` below with the actual key value. The `umask 077` ensures the file is created with mode 600 (rw for owner only). The `printf "%s\n"` appends exactly one newline.
```bash
sshpass -p 'assistant' ssh -o StrictHostKeyChecking=accept-new \
pi@192.168.50.115 \
'umask 077 && printf "%s\n" "sk-PASTE-YOUR-KEY-HERE" > ~/.openai_key'
```
- [ ] **Step 3: Verify file exists and is mode 600**
```bash
sshpass -p 'assistant' ssh pi@192.168.50.115 \
'ls -l ~/.openai_key && wc -c ~/.openai_key'
```
Expected output (mode `-rw-------`, owner `pi`, size > 50 bytes):
```
-rw------- 1 pi pi <size> <date> /home/pi/.openai_key
<size> /home/pi/.openai_key
```
If mode is not `-rw-------`, fix with `chmod 600 ~/.openai_key`.
If size is suspiciously small (< 20 bytes), the key wasn't pasted correctly — re-do step 2.
(No commit — this step modifies state on the Pi, not the repo.)
---
### Task 5: Hardware verification on the Pi
This task is the actual pass/fail gate. No automated tests; verification is the human ear + console output. If anything in this task fails, do NOT mark the implementation complete — investigate and either fix in code or update the spec to reflect what actually works.
**Files:** (none — this is a runtime verification task)
- [ ] **Step 1: Confirm the Pi is reachable**
Run from workstation:
```bash
ping -c 2 192.168.50.115
```
Expected: 2/2 replies. If unreachable, stop and ask the user.
- [ ] **Step 2: Deploy + run the probe**
Run from workstation:
```bash
./bin/probe-cs-4a
```
Expected console output (abridged, with elision):
```
>> precondition: ~/.openai_key present on Pi
OK.
>> dotnet publish (linux-arm64, self-contained)
... (build output) ...
>> scp to pi@192.168.50.115:~/probe-cs-4a/ (wipe-and-replace)
>> ssh + run on Pi (Ctrl-C from this terminal stops the probe)
[ws] connecting to wss://api.openai.com/v1/realtime?model=gpt-realtime
[ws] connected
[ws] session.created sess_xxxxxxxxxxxx
[ws] session.updated — session pinned
Using device <N> ('USB Speaker Phone (LISTENAI ...)')
[beep]
Speak now.
```
At this point the probe is waiting for you to speak. Say a single short question into the USB Speaker Phone — for example: **"What's the capital of France?"** — at normal volume from ~1 m.
Continue with verification in step 3.
If the publish step errors with a runtime crash invoking `Libc.setenv`, confirm Task 2's `[DllImport]` line for `setenv` is present.
If the Pi run errors with `Cannot open shared library libportaudio.so`, the self-contained publish didn't bundle the native libs — confirm `<SelfContained>true</SelfContained>` and `<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>` in Task 1's csproj.
If `[error] WS connect failed: ... HTTP 401` appears, the API key was rejected. Re-check Task 4 step 3 and confirm the key on the Pi starts with `sk-` and the user pasted the right value.
- [ ] **Step 3: Verify the full conversational round-trip**
After saying your question in step 2, observe:
1. `[vad] speech started` appears in the console as you begin speaking.
2. `[vad] speech stopped` appears shortly after you stop speaking (within ~0.51 s, governed by `silence_duration_ms = 500` in session.update).
3. `[ws] input_audio_buffer.committed` and `[ws] response.created` follow.
4. `[audio] first delta received (t = X.XX s since speech_stopped)` appears with a small latency value (~12 s).
5. The assistant's spoken reply comes out of the USB Speaker Phone speaker — clearly, intelligibly, recognisable as English speech.
6. `[ws] response.audio.done` and `[ws] response.done` follow.
7. `[transcript] <the assistant's reply, as text>` appears — should match what you heard.
8. `[ws] closing` and `bye` appear; the probe exits.
**Pass criterion 1:** End-to-end round-trip works once. Probe exits 0.
**Pass criterion 2:** Response is intelligible (no clipping, no dropouts longer than one ~43 ms callback).
**Pass criterion 8:** Probe exits within ~500 ms of the last audio sample being heard.
- [ ] **Step 4: Check latency from the log line**
Look at the `[audio] first delta received (t = X.XX s since speech_stopped)` line printed in step 3.
**Pass criterion 3:** `X.XX` is under ~2 s. Slower is acceptable if logged here; over ~5 s is a fail.
- [ ] **Step 5: Scan for warnings**
Re-read the full console output. Confirm there are **no** lines of these forms during normal operation (between `[beep]` and `[ws] response.done`):
- `[status] upstream behind, dropping mic frame`
- `[status] input overflow`
- `[status] unexpected input frameCount=...`
- `[error] WS ...`
- `[ws] ignored type=...` (one or two of these are OK — the API may send events we don't handle; sustained occurrences during normal flow indicate a missing handler we should add)
**Pass criterion 4:** No `[status] upstream behind` warnings.
**Pass criterion 5:** No `[error] WS …` events.
**Pass criterion 6:** No PortAudio `InputOverflow` status flags.
- [ ] **Step 6: Test Ctrl-C clean exit (separate run)**
Restart the probe:
```bash
./bin/probe-cs-4a
```
When `Speak now.` appears, press Ctrl-C in the workstation terminal **without** speaking.
Expected:
```
^C
[ws] closing
bye
$
```
Verify with `echo $?` — exit code 0.
**Pass criterion 7:** Ctrl-C produces a clean exit (no stack trace, terminal returns to prompt, exit code 0 when invoked during the steady listening phase).
- [ ] **Step 7: Update `findings.md` with the outcome**
Add a new section `## Test 4a outcome (2026-06-12 — OpenAI Realtime one-shot voice round-trip in C#)` at the bottom of `findings.md` summarising:
- Pass / fail per criterion (18).
- Latency measured (the `X.XX` value from step 4).
- Any new gotchas discovered during implementation or hardware verification — especially:
- Any OpenAI Realtime API surface differences from the spec (event types not in our switch, session.update fields rejected, model name issues).
- Any audio-format quirks (24 kHz playback dropouts, sample-rate-mismatch artifacts).
- Any concurrency surprises (channel deadlocks, send-loop wedges).
- Any errors in our LoadApiKey / mode-600 check.
- Implications for 4b (what we still need to debug there vs. what 4a proved out).
- Reference paths (spec + plan + code + deploy script).
Use the existing Test 1, 2, and 3 outcome sections in `findings.md` as the template.
- [ ] **Step 8: Commit the findings update**
```bash
git add findings.md
git commit -m "Findings: Test 4a OpenAI Realtime one-shot voice round-trip outcome"
```
- [ ] **Step 9: Final status report**
Report to the user:
- Pass/fail per criterion (1 through 8 from spec).
- Any criteria that failed and what was discovered.
- The measured round-trip latency value.
- A one-line summary of whether Test 4a unblocks Test 4b (i.e. whether the OpenAI Realtime surface + audio plumbing is proven, so 4b can focus on wakeword + loop + 16↔24 kHz coexistence).
---
## Self-review notes
- **Spec coverage:**
- Goal (single-binary one-shot probe) → Tasks 1+2+3+5.
- `~/.openai_key` precondition + file-on-Pi storage → Task 3 (script) + Task 4 (one-time setup) + Task 2 (`LoadApiKey` with mode-600 warning).
- WS connect with `Authorization` + `OpenAI-Beta` headers → Task 2 (`ws.Options.SetRequestHeader` + `ConnectAsync`).
- `session.update` payload with `server_vad` → Task 2 (`SendSessionUpdate` helper, exact JSON shape).
- Mic input stream at 24 kHz with `_micArmed` gate → Task 2 (`OpenInputStream`, `state.MicArmed` check in callback).
- UpstreamSendLoop (base64 + `input_audio_buffer.append`) → Task 2 (`UpstreamSendLoop`).
- ReceiveLoop with `JsonDocument` switch on `type` → Task 2 (`ReceiveLoop`, complete handler table).
- Output stream drain-or-zero-fill + `_noMoreDeltas` + `Reader.Completion.IsCompleted` completion → Task 2 (`OpenOutputStream` callback).
- Startup beep through `downstreamChannel` → Task 2 (lines after `outputStream.Start()`).
- `ManualResetEventSlim doneFlag` + clean shutdown via `finally` → Task 2 (try/finally block).
- Ctrl-C → `Console.CancelKeyPress``cts.Cancel``OperationCanceledException` → finally → exit 0 → Tasks 2 + 5 step 6.
- `bin/probe-cs-4a` with ssh -t + wipe-and-replace + key precondition → Task 3.
- Pass criteria 18 → Task 5 steps 3, 4, 5, 6.
- Findings update → Task 5 steps 78.
- **Placeholder scan:** no TBD/TODO; every code step shows the actual code; every command shows expected output. The single user-substitution placeholder is `sk-PASTE-YOUR-KEY-HERE` in Task 4 step 2, which is explicitly called out as a replacement target.
- **Type consistency:**
- `SharedState` fields (`MicArmed`, `StopSending`, `NoMoreDeltas`, `CurrentChunk`, `ChunkOffset`) referenced consistently from `OpenInputStream`, `OpenOutputStream`, `UpstreamSendLoop`, `ReceiveLoop`, and the top-level statements.
- `Channel<short[]>` used for both upstream and downstream; the `short[]` element type is consistent everywhere.
- `ManualResetEventSlim doneFlag` passed by reference into `OpenOutputStream` so the callback can `Set()` it; main thread awaits it via `Wait(cts.Token)`.
- `CancellationTokenSource cts` is the single source of cancellation across all three loops + the main thread.
- **Scope:** the spec is a single subsystem (one probe). One plan is the right granularity.
@@ -0,0 +1,254 @@
# Test 3 — full-cycle C# probe (wakeword + record + playback)
Combine the C# Test 1 (record + play) and Test 2 (wakeword + beep) into a single state machine that exercises the full assistant conversation cycle on the Pi:
> Idle, listen for "alexa" → beep → record 5 s → play the recording back through the same USB Speaker Phone → idle, listen for "alexa" again. Repeat indefinitely until Ctrl-C.
This is the last throwaway probe before the main Pi-client code starts. Its job is to surface anything the prior two probes couldn't, in particular **whether the wakeword detector and the recording/playback path coexist correctly on one USB device with one persistent input stream**.
Read `findings.md` (§ "C# probe outcome", § "C# wakeword probe outcome") before reading this. Everything below assumes those two outcomes as ground truth.
## Goal
A single-binary C# probe that:
1. Opens one persistent PortAudio input stream and one `WakewordModel` for the lifetime of the process.
2. Runs a four-state machine on each incoming 80 ms audio frame: **IDLE → BEEPING → RECORDING → PLAYBACK → IDLE**.
3. Plays a 200 ms beep on detection (fire-and-forget output stream).
4. Records 5 s of audio into an in-memory `short[]` buffer.
5. Plays that buffer back through the same USB device (fire-and-forget output stream).
6. Resets the wakeword model on every IDLE re-entry so the next "alexa" is detected quickly (target ≤ ~2 s after playback ends; see "Why detector gating works"), not garbage from stale ring buffers.
7. Exits cleanly on Ctrl-C from the workstation (SIGINT propagated via `ssh -t`).
## Non-goals
- No automated tests. Verification is hardware-only (run on Pi, judge by ear + console output).
- No software AEC. The detector is gated during PLAYBACK (state-machine discards frames); that is sufficient to prevent the playback re-triggering the wakeword. The Anhui LISTENAI USB device may have hardware AEC; not in scope to probe.
- No custom wakeword. Stock "alexa" classifier carried forward from Test 2.
- No WAV output. Test 1 wrote `out.wav`; Test 3 keeps the recording in memory. If a future debugging session needs raw captures, add an opt-in flag then.
- No `WakewordModel` extraction into a shared library. The file is copied verbatim from Test 2 into Test 3, and `Reset()` is added only to the Test 3 copy. Test 2's copy is frozen.
- No barge-in (user interrupting playback by speaking). The detector is fully gated during PLAYBACK; there is no way for the user to retrigger mid-playback.
## Architecture
```
┌──────────────────────────────────────────┐
│ PortAudio input stream (persistent) │
│ 16 kHz mono Int16, 1280 frames/cb │
│ callback memcpy → BlockingCollection │
└────────────────┬─────────────────────────┘
│ short[1280] frames
┌──────────────────────────────────────────┐
│ Consumer thread │
│ while (!cts.IsCancellationRequested) { │
│ frame = queue.Take(cts.Token); │
│ switch (_state) { │
│ case IDLE → model.Predict(...) │
│ case BEEPING → discard + count │
│ case RECORDING→ memcpy → recordBuf │
│ case PLAYBACK → discard │
│ } │
│ } │
└────────────────┬─────────────────────────┘
│ on demand
┌──────────────┴──────────────┐
▼ ▼
FireAndForgetBeep FireAndForgetPlayback
(200 ms 880 Hz tone) (5.04 s of recordBuf)
Output stream pattern from Test 2 (Start under
try/catch BEFORE Task.Run + 2 s timeout on done.Wait)
```
One process. One persistent input stream. Per-cycle output streams spawned fire-and-forget. State machine lives entirely in the consumer thread, which is the single writer to `_state`, so `_state` only needs `volatile` — no locks.
## State machine
Frame size is 1280 samples = 80 ms at 16 kHz. All counters below are in frames.
| State | Per-frame action | Transition |
|-|-|-|
| **IDLE** | `score = model.Predict(frame)` | `score ≥ 0.5` AND cooldown elapsed → log `DETECTED`, call `FireAndForgetBeep(...)`, set `beepFrames = 0`, → **BEEPING** |
| **BEEPING** | discard frame; `beepFrames++` | after **3 frames** (240 ms) → allocate `recordBuf = new short[63 * 1280]`, `recordOffset = 0`, → **RECORDING** |
| **RECORDING** | `Buffer.MemoryCopy` frame into `recordBuf` at `recordOffset`; `recordOffset += 1280` | after **63 frames** (5.04 s, full buffer) → call `FireAndForgetPlayback(recordBuf, …)`, → **PLAYBACK** |
| **PLAYBACK** | discard frame; check `volatile bool _playbackDoneFlag` | `_playbackDoneFlag == true` → reset flag, call `model.Reset()`, → **IDLE** |
Rationale for the constants:
- **BEEPING = 3 frames (240 ms).** Beep duration is 200 ms; 3 × 80 ms = 240 ms is the next frame boundary at-or-above 200 ms, leaving a 40 ms margin to absorb device buffering, speaker reverb, and PortAudio's output latency so the recording doesn't open with the tail of the beep. If verification shows audible beep tail still leaking into the recording, raise to 4 frames (320 ms) — easy tweak.
- **RECORDING = 63 frames (5.04 s).** 5 s = 62.5 frames; rounding up to 63 avoids partial-frame handling. The extra 40 ms of recording is inaudibly different from 5 s.
- **Cooldown** applies only to the IDLE→BEEPING transition. The state machine itself blocks re-detection in the other states. 1 s cooldown carries forward from Test 2.
`_playbackDoneFlag` is set by the playback output stream's audio callback when its read offset reaches the end of `recordBuf`. The consumer checks the flag on every PLAYBACK frame (and only on PLAYBACK frames). Between the flag firing and the consumer seeing it, the consumer continues to drain input frames into the discard branch — that's what keeps the input stream from backing up.
## Why detector gating works (and why Reset matters)
The USB Speaker Phone is the same physical device for input and output. During PLAYBACK, the recorded audio comes out the speaker and re-enters the input stream. If `WakewordModel.Predict` were running across all states, any "alexa" buried in the recording would re-trigger detection, kicking the state machine back into BEEPING mid-playback — an infinite loop.
The state-machine gating (PLAYBACK discards every frame, doesn't call `Predict`) prevents the re-trigger directly. That much is straightforward.
The subtler issue is what happens to `WakewordModel`'s internal ring buffers (`_rawRing`, `_melRing`, `_embRing`) during BEEPING + RECORDING + PLAYBACK. With ~10 s of skipped `Predict` calls (240 ms BEEPING + 5.04 s RECORDING + ~5 s PLAYBACK), the rings hold audio from ~10 s ago — pre-detection. When we return to IDLE and resume `Predict`, the next call computes mel features from a window that straddles 10-second-old audio at the bottom and fresh post-playback audio at the top. The first one or two predictions on re-entry would be on this garbage window.
The `WarmupFrames = 16` guard already in `WakewordModel.cs` exists to suppress exactly this kind of cold-start junk on initial process boot. It does NOT re-arm after a pause. So without intervention, the first ~1.3 s after returning to IDLE would be a false-positive vulnerability window.
**Fix:** Add a `WakewordModel.Reset()` method that clears `_rawRingFill = 0`, `_melRing.Clear()`, `_embRing.Clear()`, `_framesSeen = 0`. The consumer calls it on the PLAYBACK→IDLE transition. The existing `WarmupFrames` guard then suppresses the next 16 `Predict` calls (~1.3 s of audio) until the rings refill with fresh post-playback audio. After that, detection runs as it did at startup.
This means the next "alexa" after playback can only be detected on a frame at least 1.3 s after the IDLE re-entry, plus whatever time the user takes to start saying "alexa". The verification budget for "next detection within ~1 s of playback" needs to acknowledge this: the practical lower bound is ~1.3 s (warmup) + ~0.5 s (user reaction + saying the word). See "Pass criteria" below for the adjusted threshold.
## Components and files
```
tests/03-full-cycle-cs/
├── Probe.csproj
│ <TargetFramework>net9.0</TargetFramework>
│ <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
│ <SelfContained>true</SelfContained>
│ <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
│ <InvariantGlobalization>true</InvariantGlobalization>
│ PackageReference PortAudioSharp2 1.0.6
│ PackageReference Microsoft.ML.OnnxRuntime 1.26.0
│ Content Include="models/*.onnx" CopyToOutputDirectory=PreserveNewest
│ RootNamespace=FullCycleProbe, AssemblyName=Probe
├── Program.cs ~200 lines, top-level statements:
│ 1. Environment.SetEnvironmentVariable + Libc.setenv for
│ PA_ALSA_PLUGHW=1 and ORT_LOGGING_LEVEL=3
│ 2. OrtEnv.CreateInstanceWithOptions with
│ logLevel = ORT_LOGGING_LEVEL_ERROR (BEFORE any InferenceSession)
│ 3. PortAudio.Initialize, FindUsbDevice (lifted from Test 2)
│ 4. WakewordModel ctor with models/{melspectrogram,embedding_model,alexa}.onnx
│ 5. BlockingCollection<short[]> queue (bounded 16)
│ 6. CancellationTokenSource + Console.CancelKeyPress handler
│ 7. Input StreamParameters + callback (memcpy → queue.TryAdd)
│ 8. Stream construction + Start
│ 9. Consumer loop with state-machine switch (sketched above)
│ 10. finally: stream.Stop, model.Dispose, PortAudio.Terminate
│ Helpers: FireAndForgetBeep (from Test 2), FireAndForgetPlayback (new,
│ same shape but reads recordBuf and sets _playbackDoneFlag),
│ MakeBeep, FindUsbDevice, static class Libc.setenv
├── WakewordModel.cs Copied verbatim from tests/02-wakeword-cs/.
│ Add Reset() method (≈5 lines):
│ public void Reset() {
│ _rawRingFill = 0;
│ _melRing.Clear();
│ _embRing.Clear();
│ _framesSeen = 0;
│ }
│ Namespace changes from WakewordProbe → FullCycleProbe.
└── models/ Copied from tests/02-wakeword-cs/models/.
melspectrogram.onnx, embedding_model.onnx, alexa.onnx
SHA-256 unchanged from the vendored set in commit 849ad47.
bin/probe-cs-3 Copy of bin/probe-cs-2 with two string swaps:
tests/02-wakeword-cs → tests/03-full-cycle-cs
~/probe-cs-2 → ~/probe-cs-3
PUBLISH_DIR /tmp/probe-cs-2-out → /tmp/probe-cs-3-out
ssh -t is retained (Ctrl-C propagation).
chmod +x at install time.
```
## Concurrency model
Three threads of execution in steady state:
1. **PortAudio input audio thread** (managed by the library). Runs the input callback every 80 ms. Allocates a `short[1280]`, memcpy's the input buffer, calls `queue.TryAdd(frame, 0)`. On `TryAdd` failure (queue full), logs `[status] consumer behind, dropping frame`. On `InputOverflow` status flag, logs `[status] input overflow`. Never blocks.
2. **Consumer thread** (`Task.Run` or just main thread after `stream.Start`). The state-machine loop above. Single writer of `_state`. Calls `model.Predict` in IDLE; calls `model.Reset` on PLAYBACK→IDLE.
3. **PortAudio output audio threads** (one per fire-and-forget beep/playback). Each output stream has its own callback that writes from a managed buffer (beep buffer or recordBuf). Sets `done.Set()` (beep) or `_playbackDoneFlag = true` (playback) when its source buffer is fully consumed.
Shared mutable state:
- `volatile State _state` — single writer (consumer), multiple readers (consumer only, actually). Volatile is enough.
- `volatile bool _playbackDoneFlag` — set by playback audio thread, read+reset by consumer.
- `BlockingCollection<short[]> queue` — its own concurrency.
No locks. No `Interlocked`. The state machine is intentionally a single-writer design so we don't have to reason about race conditions.
## Error handling
- **Input overflow** (PortAudio callback status flag): log `[status] input overflow`, continue. No corrective action — the buffer dropped audio at the kernel level, nothing the consumer can do.
- **Consumer-behind** (`queue.TryAdd` returns false because the bounded queue is full): log `[status] consumer behind, dropping frame`, continue. In steady state this should never fire; if it does during verification it's a signal that one of the state branches is too slow (likely `Predict` exceeding 80 ms, which Test 2 confirmed doesn't happen on the Pi).
- **Output stream `Start()` throws**: dispose the stream, log `[error] output stream failed to start: <ex>`, snap `_state` back to IDLE, call `model.Reset()`. The current cycle is lost; the probe survives. (Test 2 surfaced this as the exact failure mode the fire-and-forget pattern guards against.)
- **`WakewordModel` exceptions** during `Predict` or `Reset`: let propagate. The `finally` block disposes the model and terminates PortAudio. Probe exits non-zero. This is fail-fast on the assumption that any model exception is an invariant violation worth investigating.
- **Ctrl-C from workstation**: `Console.CancelKeyPress` calls `cts.Cancel()`. The consumer's `queue.Take(cts.Token)` throws `OperationCanceledException`, the outer `try` swallows it, the `finally` block runs `stream.Stop` + `model.Dispose` + `PortAudio.Terminate`. Process exits 0. `ssh -t` in `bin/probe-cs-3` ensures the SIGINT reaches the remote process.
## Gotchas carried forward (from findings.md)
1. **Libc.setenv for env vars libc reads via getenv().** `Environment.SetEnvironmentVariable` does NOT propagate to libc on Linux. Apply `Libc.setenv` for both `PA_ALSA_PLUGHW` (PortAudio) and `ORT_LOGGING_LEVEL` (onnxruntime) alongside the managed call.
2. **OrtEnv.CreateInstanceWithOptions BEFORE first InferenceSession.** ORT 1.26.0's GPU-detection logger fires from C++ before managed code runs, so `Libc.setenv("ORT_LOGGING_LEVEL", ...)` alone doesn't suppress the `/sys/class/drm/card0` warnings. Construct `OrtEnv` with `logLevel = ORT_LOGGING_LEVEL_ERROR` *before* `WakewordModel`'s constructor runs.
3. **`using Stream = PortAudioSharp.Stream;`** at the top of `Program.cs` to disambiguate from `System.IO.Stream` that `ImplicitUsings` pulls in.
4. **`<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`** in `Probe.csproj` for the callback's `Buffer.MemoryCopy` over `short*`.
5. **Fire-and-forget output stream lifetime.** `stream.Start()` runs synchronously under try/catch *before* `Task.Run` launches; on `Start()` failure, dispose + rethrow without ever creating the cleanup task. The cleanup task waits on `done.Wait(TimeSpan.FromSeconds(2))` — 2 s timeout so a faulting callback that silently fails (PortAudio swallows callback exceptions) can't hang cleanup. Applied to both `FireAndForgetBeep` AND `FireAndForgetPlayback`.
6. **Vendored alexa classifier has static batch dim `[1, 16, 96]`.** `WakewordModel`'s shape assertion intentionally skips dim 0 so both `1` (static) and `-1` (dynamic) pass. Preserve.
7. **Mel-spectrogram output shape `(1, 1, n_frames, 32)`.** `n_frames` lives at `Dimensions[2]`, not `[1]`. Preserve in the copied `WakewordModel.cs`.
8. **`ssh -t` for Ctrl-C propagation.** `bin/probe-cs-3` forces a pseudo-TTY so workstation SIGINT reaches the remote `./Probe` and triggers `Console.CancelKeyPress`.
## Verification (hardware only, no automated tests)
Run from workstation:
```
./bin/probe-cs-3
```
Expected console flow (abridged):
```
[wakeword-model] melspectrogram.onnx
input 'input': shape=[1,-1] dtype=Single
output 'output': shape=[1,1,-1,32] dtype=Single
[wakeword-model] embedding_model.onnx
...
[wakeword-model] alexa.onnx
input 'onnx::Unsqueeze_0': shape=[1,16,96] dtype=Single
output ...
Loaded in 0.6s.
Using device 3 ('USB Speaker Phone (LISTENAI ...)')
Listening. Say 'alexa'. Ctrl-C to exit.
DETECTED alexa score=0.872 t=4.3s
[state] BEEPING
[state] RECORDING
[state] PLAYBACK
[state] IDLE
DETECTED alexa score=0.811 t=15.2s
...
```
(Exact `[wakeword-model]` shape output reflects whatever the ONNX models actually report; the example above is illustrative.)
### Pass criteria
All must hold during a single run on the Pi:
1. **3 full cycles back-to-back** without restarting the probe. Each playback is clearly recognisable as what was spoken into the mic in the preceding 5 s.
2. **Next "alexa" after playback** is detected within ~2 s of playback ending. (The ~1.3 s `WarmupFrames` re-arm after `model.Reset()` is a hard floor; budget another ~0.5 s for user reaction + saying the word. If detection consistently takes longer than ~2 s, that's a regression vs. Test 2's IDLE-state detection latency and worth investigating.)
3. **Wakeword detection rate during IDLE** ≥ 8/10 attempts (carries forward from Test 2's measured rate).
4. **CPU during IDLE steady state** < 50% of one core, measured by `top` on the Pi. (Test 2 measured 1731% on the same workload; Test 3 adds only a state-machine switch — no extra CPU in IDLE.)
5. **No `[status] input overflow` or `[status] consumer behind` warnings** during any phase of normal operation.
6. **No stuck states.** The probe never sits in BEEPING, RECORDING, or PLAYBACK for longer than its bounded duration. (BEEPING ≤ 240 ms, RECORDING ≤ 5.04 s, PLAYBACK ≤ ~5.5 s including device drain.)
7. **Workstation Ctrl-C** triggers a clean exit: `\nbye` line, process exit 0, no stack trace, terminal returns to prompt.
### What success teaches us for the main assistant
- The "persistent input stream + state-machine consumer + per-cycle output streams" shape works on this hardware. The main assistant can build directly on it.
- Detector gating during playback (without AEC) is sufficient to prevent self-trigger, at least with stock "alexa" and the recorded user audio. Real TTS may need re-evaluation; this probe doesn't answer that.
- `WakewordModel.Reset()` belongs in the production model interface — the main assistant will need it for the same reasons.
### What failure would tell us
- If detection rate drops below 8/10 in IDLE: the state-machine overhead is interfering. Investigate (unlikely — the switch is O(constant)).
- If `[status] consumer behind` warnings appear: one of the consumer branches is too slow. Most likely `Predict` exceeded 80 ms (Test 2 didn't measure this directly). Add per-`Predict` timing and re-run.
- If playback re-triggers detection (probe loops in BEEPING/RECORDING/PLAYBACK indefinitely): `Reset` is wrong, or `_playbackDoneFlag` plumbing has a race. Investigate via logging which state the trigger occurred in.
- If `Ctrl-C` produces a stack trace: `OperationCanceledException` isn't being caught at the right level. Fix.
## Reference paths
- Spec (this doc): `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`
- Plan (next): `docs/superpowers/plans/2026-06-12-test-3-full-cycle-csharp.md`
- Code (to be written): `tests/03-full-cycle-cs/`
- Deploy (to be written): `bin/probe-cs-3`
- Test 1 reference: `tests/01-record-play-cs/Program.cs`, `tests/01-record-play-cs/WavWriter.cs`, `bin/probe-cs`
- Test 2 reference: `tests/02-wakeword-cs/Program.cs`, `tests/02-wakeword-cs/WakewordModel.cs`, `bin/probe-cs-2`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
- Canonical findings: `findings.md` § "C# probe outcome (2026-06-12 — Test 1 ported to C#)" and § "C# wakeword probe outcome (2026-06-12 — Test 2 ported to C#)"
@@ -0,0 +1,474 @@
# Test 4a — OpenAI Realtime API one-shot voice round-trip (C#)
A throwaway C# probe that has a single round-trip spoken conversation with the OpenAI Realtime API:
> Boot → beep → user speaks one sentence → server VAD ends the turn → assistant audio reply plays back through the same USB Speaker Phone → exit 0.
No wakeword. No loop. No state machine. One process, runs once, dies.
Test 4a's job is to debug the OpenAI Realtime API surface (auth, WS handshake, `session.update` payload, event JSON shapes, base64 audio decode, server VAD endpointing, streaming-length output-stream cleanup) **in isolation** from the harder concurrency story of Test 4b (wakeword + loop + 16↔24 kHz coexistence).
Read `findings.md` (§ "C# probe outcome", § "C# wakeword probe outcome", § "C# full-cycle outcome") before reading this. Everything below assumes those outcomes as ground truth. The patterns lifted directly from Test 3 are: `Libc.setenv`, the `using Stream = PortAudioSharp.Stream;` import disambiguation, `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`, the fire-and-forget output stream lifetime (Start under try/catch BEFORE the cleanup task launches), `FindUsbDevice`, the `MakeBeep` helper, `bin/probe-cs-3`'s wipe-and-replace publish shape with `ssh -t`.
## Goal
A single-binary C# probe that:
1. Reads the OpenAI API key from `~/.openai_key` on the Pi (mode 600).
2. Connects to `wss://api.openai.com/v1/realtime?model=gpt-realtime` over a WebSocket with `Authorization: Bearer …` + `OpenAI-Beta: realtime=v1` headers.
3. Sends one `session.update` event pinning audio formats, voice, instructions, and `turn_detection: server_vad`.
4. Plays a startup beep through the USB Speaker Phone.
5. Opens a 24 kHz mono Int16 input stream on the same device; streams mic audio to OpenAI as `input_audio_buffer.append` events.
6. Waits for the server VAD to fire `input_audio_buffer.speech_stopped` (server then auto-commits the buffer and creates the response).
7. Stops sending mic audio at that point. Receives `response.audio.delta` chunks, base64-decodes them, plays them through a 24 kHz output stream on the same device.
8. On `response.done` AND the output buffer drained, closes the WebSocket cleanly, terminates PortAudio, exits 0.
9. Exits cleanly on Ctrl-C from the workstation (SIGINT propagated via `ssh -t`) at any stage.
## Non-goals
- **No wakeword.** No `WakewordModel.cs`. No 16 kHz audio path anywhere in 4a. (Test 4b layers that on top of this proven base.)
- **No state machine, no loop.** Probe runs once and exits.
- **No barge-in.** Once `speech_stopped` fires, mic frames are dropped on the floor. The user cannot interrupt the assistant in 4a.
- **No client-side resampling.** Mic = 24 kHz, output = 24 kHz, OpenAI = 24 kHz pcm16. PortAudio plug-resamples 48↔24 at ALSA on the USB device.
- **No WAV capture.** Logging is stdout/stderr only.
- **No retry on WS errors, no reconnect, no health check.** Fail loud, exit non-zero, log enough to diagnose.
- **No automated tests.** Hardware verification only.
## Architecture
```
┌──────────────────────────────────────┐
│ ClientWebSocket │
│ wss://api.openai.com/v1/realtime? │
│ model=gpt-realtime │
│ Headers: Authorization: Bearer … │
│ OpenAI-Beta: realtime=v1 │
└────┬──────────────────────────┬──────┘
│ SendAsync │ ReceiveAsync
▼ ▼
┌──────────────────────────┐ ┌──────────────────────────────┐
│ UpstreamSendLoop │ │ ReceiveLoop (Task.Run) │
│ (Task.Run) │ │ while (!cts.Cancel) { │
│ drain upstreamChannel: │ │ msg = ReceiveAsync(…) │
│ pull short[1920] │ │ doc = JsonDocument.Parse │
│ → base64 │ │ switch (doc["type"]) │
│ → input_audio_ │ │ session.created → log │
│ buffer.append │ │ session.updated → log │
│ → ws.SendAsync │ │ speech_started → log │
│ exits when │ │ speech_stopped → │
│ _stopSending && empty│ │ _stopSending = true │
└──────────┬───────────────┘ │ response.audio.delta: │
│ (FullMode.Wait) │ base64 → short[] │
│ │ → downstreamChannel │
▲ │ .Writer.WriteAsync │
│ short[1920] │ audio_transcript.delta:│
│ (40 ms @ 24 kHz) │ append to log buf │
┌──────────┴───────────────┐ │ response.done → │
│ PortAudio input stream │ │ _noMoreDeltas = true │
│ 24 kHz mono Int16, │ │ downstreamChannel │
│ 1920 frames/cb │ │ .Writer.Complete │
│ callback: │ │ error → log + Cancel │
│ if !_micArmed: discard│ └────────────┬─────────────────┘
│ elif _stopSending: │ │
│ discard │ ▼
│ else: │ ┌──────────────────────────────┐
│ copy → short[] │ │ Channel<short[]> │
│ TryWrite to │ │ (bounded cap 64, │
│ upstreamChannel │ │ FullMode.Wait) │
└──────────────────────────┘ └────────────┬─────────────────┘
┌──────────────────────────────────┐
│ PortAudio output stream │
│ 24 kHz mono Int16, │
│ 1024 frames/cb │
│ callback: │
│ fill = 0 │
│ while fill < frameCount: │
│ if currentChunk == null: │
│ try pull short[] or break │
│ take = min(remaining, slack)│
│ memcpy chunk → output │
│ zero-fill remaining │
│ if currentChunk == null │
│ && Reader.Count == 0 │
│ && Completion.IsCompleted: │
│ doneFlag.Set() │
│ return Complete │
│ return Continue │
└────────────┬─────────────────────┘
main thread: doneFlag.Wait
→ ws.CloseAsync(NormalClosure)
→ input/output stream.Stop
→ PortAudio.Terminate
→ exit 0
```
One process. Three managed threads (main + UpstreamSendLoop + ReceiveLoop) plus PortAudio's input and output audio threads (managed by the library).
**Shared mutable state — the full list, small on purpose:**
- `Channel<short[]> upstreamChannel`, bounded 64, `FullMode.Wait`, single-writer (mic callback), single-reader (UpstreamSendLoop). Holds mic frames waiting to be base64'd and sent.
- `Channel<short[]> downstreamChannel`, bounded 64, `FullMode.Wait`, **multi-writer** (main thread writes the startup beep; ReceiveLoop writes response audio chunks; temporally separated but Channel<T>'s `SingleWriter` flag is a contract not a runtime check, so keep it `false`), single-reader (output callback). Holds decoded audio chunks waiting to be played. Its `Writer.Complete()` (called by ReceiveLoop on `response.done`) is the WS-stream-end signal for the output callback.
- `volatile bool _micArmed`: starts `false`; set `true` after the startup beep has been queued and drained. Mic frames before this point are discarded so the beep doesn't get captured and shipped to OpenAI.
- `volatile bool _stopSending`: set by ReceiveLoop on `input_audio_buffer.speech_stopped`. Read by both the mic callback (which then discards) and UpstreamSendLoop (which then drains its queue and exits).
- `volatile bool _noMoreDeltas`: set by ReceiveLoop on `response.done`. Not strictly required because `downstreamChannel.Writer.Complete()` carries the same signal — but kept as a redundant fast path that the output callback can check without touching the channel.
- `ManualResetEventSlim doneFlag`: set by the output callback when it has drained the channel after completion. Awaited by main.
- `CancellationTokenSource cts`: wired to `Console.CancelKeyPress` + any WS error. Cancels both Task.Run loops; main's `doneFlag.Wait(cts.Token)` honours it.
No locks. No `Interlocked`. The bounded `Channel<T>` handles all the cross-thread audio handoff with the right backpressure semantics.
## WebSocket lifecycle
Six discrete phases, all in `Program.cs` top-level statements + the two `Task.Run` loops:
1. **Load API key.** `LoadApiKey()` helper reads `~/.openai_key`, trims whitespace. Mode 600 verified — non-fatal warning if looser. Fatal error (exit 2) if the file is missing, empty, or doesn't start with `sk-`.
2. **Construct `ClientWebSocket`** with two request headers: `Authorization: Bearer <key>` and `OpenAI-Beta: realtime=v1`. `ConnectAsync` to `wss://api.openai.com/v1/realtime?model=gpt-realtime` with `cts.Token`. On failure (DNS, TLS, 401, 403, 429, 5xx), log the exception message + any HTTP status carried on `WebSocketException.WebSocketErrorCode` / inner `HttpRequestException.StatusCode`; exit 3.
3. **Launch ReceiveLoop** (`Task.Run`) before sending anything, so we don't miss `session.created`.
4. **Send `session.update`.** First client message; pins the session config (see "Client → server events" below). Don't strictly block on `session.updated` — the receive loop logs it asynchronously — but the spec expects it within ~200 ms.
5. **Open + start PortAudio streams.** Input stream first (24 kHz mono Int16, 1920 frames/cb), then output stream (24 kHz mono Int16, 1024 frames/cb). Both opened on `FindUsbDevice()`. Launch UpstreamSendLoop (`Task.Run`).
6. **Startup beep + arm mic.** Queue a 200 ms 880 Hz beep (`MakeBeep` lifted from Test 3, regenerated at 24 kHz) onto `downstreamChannel`. Sleep ~300 ms (beep + drain margin). Set `_micArmed = true`. Print `Speak now.` to stdout.
7. **Wait for `doneFlag`** on the main thread (with `cts.Token` for Ctrl-C). On wake-up: `ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", …)` with a 2 s timeout token. Stop input stream, stop output stream, terminate PortAudio, exit 0.
## Client → server event payloads
Three send-once messages on the main thread, plus a stream of `input_audio_buffer.append` from UpstreamSendLoop. All written with `JsonSerializer.Serialize` over `Dictionary<string, object?>` (lightest thing that doesn't require typed records for a probe).
```json
// (1) Pinned session config — first thing sent after WS connect.
{
"type": "session.update",
"session": {
"modalities": ["audio", "text"],
"instructions": "Reply in one short sentence.",
"voice": "alloy",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"prefix_padding_ms": 300,
"silence_duration_ms": 500,
"create_response": true
}
}
}
// (2) Per mic frame (~40 ms cadence), from UpstreamSendLoop:
{ "type": "input_audio_buffer.append", "audio": "<base64-pcm16-24k>" }
```
No `conversation.item.create` and no `response.create` from the client — server VAD's `create_response: true` makes the server commit and create the response automatically on `speech_stopped`.
`instructions: "Reply in one short sentence."` keeps the response audio bounded (~2-4 s of speech) so a probe run is fast to verify by ear and cheap on quota.
`voice: "alloy"` is the long-standing default voice and is documented as universally available for raw WS sessions.
## Server → client event handlers
ReceiveLoop uses a 4 KB receive buffer + standard `ReceiveAsync` loop with `EndOfMessage` accumulation into a `MemoryStream`. On message end, `JsonDocument.Parse`. Switch on the top-level `type` property's string value:
| `type` | Action |
|-|-|
| `session.created` | log `[ws] session.created <session_id>` |
| `session.updated` | log `[ws] session.updated — session pinned` |
| `input_audio_buffer.speech_started` | log `[vad] speech started` |
| `input_audio_buffer.speech_stopped` | log `[vad] speech stopped`; set `_stopSending = true` |
| `input_audio_buffer.committed` | log only — server confirms it sealed the buffer |
| `response.created` | log only |
| `response.audio_transcript.delta` | append `delta` to an accumulating `StringBuilder` |
| `response.audio.delta` | base64-decode `delta` → bytes → reinterpret as `short[]``await downstreamChannel.Writer.WriteAsync(samples, cts.Token)`. The `await` is the backpressure path: if the channel is full, this blocks the receive loop, which blocks the WS receive buffer — correct behavior for a slow speaker. |
| `response.audio.done` | log only |
| `response.done` | set `_noMoreDeltas = true`; `downstreamChannel.Writer.Complete()`; print the accumulated transcript |
| `error` | log the full event JSON; `cts.Cancel()` |
| anything else | log `[ws] ignored type=<…>` once (or at debug verbosity), continue |
## Audio buffering — input
- Mic callback runs every 40 ms (1920 samples @ 24 kHz mono Int16, matches `framesPerBuffer = 1920`).
- Callback allocates `short[1920]`, `Buffer.MemoryCopy`s the input pointer, `upstreamChannel.Writer.TryWrite(frame)`.
- On TryWrite failure (channel full = UpstreamSendLoop is behind): log `[status] upstream behind, dropping mic frame`, continue.
- Discard frames when `_micArmed == false` (during startup beep) or `_stopSending == true` (after `speech_stopped`).
- On the `InputOverflow` status flag from PortAudio: log `[status] input overflow`, continue.
UpstreamSendLoop pseudocode:
```
await ReceiveLoop signals session.updated (or just optimistically loop from start)
while (!cts.IsCancellationRequested):
if _stopSending && upstreamChannel.Reader.Count == 0: break
frame = await upstreamChannel.Reader.ReadAsync(cts.Token)
if _stopSending: continue // drop frames captured after server VAD cutoff
bytes = MemoryMarshal.AsBytes(frame.AsSpan())
base64 = Convert.ToBase64String(bytes)
json = JsonSerializer.SerializeToUtf8Bytes(new {
type = "input_audio_buffer.append",
audio = base64
})
await ws.SendAsync(json, WebSocketMessageType.Text, endOfMessage: true, cts.Token)
```
## Audio buffering — output
`downstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(64) { FullMode = BoundedChannelFullMode.Wait, SingleWriter = false, SingleReader = true })`. `SingleWriter = false` because the main thread queues the startup beep AND ReceiveLoop queues response audio chunks (temporally separated, but the flag is a hard contract).
Capacity rationale: 64 chunks × ~40-80 ms per `response.audio.delta` ≈ 2.5-5 s of buffered audio. Comfortable headroom against network jitter without holding silly amounts of memory.
ReceiveLoop writes `short[]` chunks of varying length (one per `response.audio.delta` event). On `response.done`, calls `downstreamChannel.Writer.Complete()`.
Output callback's job is to fill `frameCount` shorts every callback (1024 frames = ~43 ms @ 24 kHz). It maintains a `short[]? _currentChunk` + `int _chunkOffset` across calls. Pseudocode:
```
int fill = 0;
while (fill < frameCount) {
if (_currentChunk == null) {
if (!downstreamChannel.Reader.TryRead(out _currentChunk)) break; // drained
_chunkOffset = 0;
}
int slack = frameCount - fill;
int remaining = _currentChunk.Length - _chunkOffset;
int take = Math.Min(remaining, slack);
unsafe {
fixed (short* src = &_currentChunk[_chunkOffset])
Buffer.MemoryCopy(src, output + fill, take * 2, take * 2);
}
_chunkOffset += take;
fill += take;
if (_chunkOffset >= _currentChunk.Length) _currentChunk = null;
}
// Zero-fill the unfilled tail (underrun mid-response, or post-completion drain).
for (int i = fill; i < frameCount; i++) output[i] = 0;
// Completion check: drained AND writer completed AND no working chunk left.
if (_currentChunk == null
&& downstreamChannel.Reader.Count == 0
&& downstreamChannel.Reader.Completion.IsCompleted)
{
doneFlag.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
```
**Underrun behavior: zero-fill, return Continue.** If `response.done` hasn't arrived yet and the next `response.audio.delta` is just in flight, we play silence for one callback (~43 ms) instead of starving — the stream stays armed, no click on resume.
**This is the streaming-length cleanup discipline** that replaces Test 3's "scale `done.Wait` timeout to the buffer length" trick. Test 3 knew the playback wall-clock up front (`recordBuf.Length / SampleRate`); Test 4a does not, because the assistant audio length is unknown a priori. So the signaling moves from "wait this many seconds" to "wait until (writer-completed AND channel-empty AND working-chunk-empty)." The output callback is the single oracle for this triple condition.
**Startup beep wrinkle:** the beep is queued onto `downstreamChannel` BEFORE the receive loop starts producing. The output callback drains it just like any other audio. The completion check is safe because `Writer.Complete()` is only ever called on `response.done` — the beep getting drained does NOT cause an early completion firing.
## Components and files
```
tests/04a-realtime-oneshot-cs/
├── Probe.csproj
│ <TargetFramework>net9.0</TargetFramework>
│ <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
│ <SelfContained>true</SelfContained>
│ <PublishSingleFile>false</PublishSingleFile>
│ <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
│ <InvariantGlobalization>true</InvariantGlobalization>
│ <Nullable>enable</Nullable>
│ <ImplicitUsings>enable</ImplicitUsings>
│ RootNamespace=RealtimeOneShotProbe, AssemblyName=Probe
│ PackageReference PortAudioSharp2 Version="1.0.6"
│ (no Microsoft.ML.OnnxRuntime — no wakeword in 4a)
│ System.Net.WebSockets + System.Text.Json come from the BCL; no package.
├── Program.cs ~300 lines, top-level statements:
│ 1. Libc.setenv("PA_ALSA_PLUGHW", "1", 1) + Environment mirror
│ (no ORT env vars — no onnxruntime in 4a)
│ 2. PortAudio.Initialize
│ 3. apiKey = LoadApiKey()
│ 4. var cts = new CancellationTokenSource();
│ Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
│ 5. var ws = new ClientWebSocket();
│ ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
│ ws.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");
│ await ws.ConnectAsync(uri, cts.Token);
│ 6. Channel<short[]> upstreamChannel, downstreamChannel (both bounded 64)
│ ManualResetEventSlim doneFlag = new(false);
│ 7. Task receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel,
│ cts, _stopSending, _noMoreDeltas, transcriptBuilder));
│ 8. Send session.update (JsonSerializer.SerializeToUtf8Bytes over a dict)
│ 9. int device = FindUsbDevice();
│ 10. Open + start input stream (24 kHz, 1920 frames/cb)
│ 11. Open + start output stream (24 kHz, 1024 frames/cb)
│ 12. Task sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel,
│ cts, _stopSending));
│ 13. Queue MakeBeep(880, 0.2, 24000) onto downstreamChannel; sleep 300 ms;
│ _micArmed = true; Console.WriteLine("Speak now.");
│ 14. try { doneFlag.Wait(cts.Token); }
│ catch (OperationCanceledException) { /* Ctrl-C path */ }
│ 15. finally:
│ ws.CloseAsync(NormalClosure, "done", linkedCtsWith2sTimeout.Token)
│ inputStream.Stop; outputStream.Stop;
│ inputStream.Dispose; outputStream.Dispose;
│ PortAudio.Terminate;
│ Helpers (all in Program.cs):
│ - LoadApiKey() — reads ~/.openai_key, perms-warns, sanity
│ - MicCallback — discards unless _micArmed && !_stopSending
│ - OutputCallback — drain-or-zero-fill loop above
│ - MakeBeep(hz, sec, rate) — sin wave Int16 (lifted from Test 3, retuned 24 kHz)
│ - UpstreamSendLoop(...)
│ - ReceiveLoop(...) — JsonDocument switch on "type"
│ - FindUsbDevice() — lifted from Test 3 verbatim
│ - static class Libc { setenv }
└── (no models/ dir, no WakewordModel.cs)
bin/probe-cs-4a Copy of bin/probe-cs-3 with substitutions:
tests/03-full-cycle-cs → tests/04a-realtime-oneshot-cs
~/probe-cs-3 → ~/probe-cs-4a
PUBLISH_DIR /tmp/probe-cs-3-out → /tmp/probe-cs-4a-out
ssh -t retained (Ctrl-C propagation).
Wipe-and-replace publish dir retained (partial-deploy
footgun from Test 3).
Adds ONE new precondition step before publishing:
sshpass -p "$PI_PASS" ssh "$PI_USER@$PI_HOST" \
'test -f ~/.openai_key && test -r ~/.openai_key' \
|| { echo "Create ~/.openai_key on the Pi (chmod 600),
paste your sk-... key into it, then re-run."; \
exit 1; }
This catches the silent "key missing" failure mode at
the deploy step rather than 30 s into the publish + scp.
```
## Concurrency model
Five concurrent threads of execution in steady state:
1. **Main thread.** Setup, session.update send, beep + arm, `doneFlag.Wait`, teardown via `finally`.
2. **ReceiveLoop** (`Task.Run`). Pulls WS messages, dispatches on `type`. Single writer of `downstreamChannel`, `_stopSending`, `_noMoreDeltas`, transcript builder. Calls `cts.Cancel()` on error.
3. **UpstreamSendLoop** (`Task.Run`). Pulls mic frames from `upstreamChannel`, JSON-serializes `input_audio_buffer.append`, calls `ws.SendAsync`. Single writer of the WS send stream.
4. **PortAudio input audio thread** (managed by the library). Runs the mic callback every 40 ms. Allocates `short[1920]`, memcpy's the input, `upstreamChannel.Writer.TryWrite(frame)`. Never blocks.
5. **PortAudio output audio thread** (managed by the library). Runs the output callback every 43 ms. Drains `downstreamChannel` into the output buffer; zero-fills underruns; sets `doneFlag` on completion. Never blocks.
Note that `ws.SendAsync` and `ws.ReceiveAsync` can both be active concurrently — `ClientWebSocket` supports one concurrent send + one concurrent receive (but NOT multiple concurrent sends or multiple concurrent receives). Our design uses exactly one of each, so this is safe.
## Error handling
| Failure mode | Behavior | Exit |
|-|-|-|
| `~/.openai_key` missing on Pi | `bin/probe-cs-4a` precondition check exits before publishing. | 1 |
| `~/.openai_key` exists but empty / not `sk-`-shaped | `LoadApiKey()` logs `[error] API key file empty` / `[error] API key doesn't start with sk-`, exits. | 2 |
| `ws.ConnectAsync` throws (DNS, TLS, 401, 403, 429, 500, network down) | Log `[error] WS connect failed: {ex.Message}` plus inner `HttpRequestException.StatusCode` if present. | 3 |
| Server `error` event mid-session | ReceiveLoop logs the full event JSON, `cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`; `finally` runs CloseAsync (no-throw with timeout) + stream.Stop + PortAudio.Terminate. | 4 |
| WS drops mid-conversation (`WebSocketException` in ReceiveLoop) | Same as above — log + cancel + cleanup-via-finally. | 4 |
| PortAudio input stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 5 |
| PortAudio output stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 6 |
| Server takes forever and never sends `response.done` | No timeout in 4a. User's recourse is Ctrl-C. (If verification surfaces this, add a configurable timeout in 4b — out of scope for 4a.) | 0 (via Ctrl-C) |
| Mic callback can't `TryWrite` to `upstreamChannel` (UpstreamSendLoop falling behind) | Log `[status] upstream behind, dropping mic frame`, continue. (Same shape as Test 3's "consumer behind" log.) | 0 |
| Workstation Ctrl-C | `Console.CancelKeyPress``e.Cancel = true; cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`. `finally` runs full cleanup path. | 0 |
The `finally` block at the bottom of `Program.cs`'s outer `try` runs **once** and is the single point where WS is closed, streams are stopped, and PortAudio is terminated. Cleanup is idempotent (`stream.Stop` is a no-op if already stopped; `PortAudio.Terminate` is fine to call once even if some path didn't run).
## Gotchas carried forward (from findings.md)
1. **`Libc.setenv("PA_ALSA_PLUGHW", "1", 1)`** before any PortAudio init. `Environment.SetEnvironmentVariable` alone does NOT propagate to libc on Linux. Apply alongside the managed call.
2. **No OrtEnv work in 4a** — no `Microsoft.ML.OnnxRuntime` dependency. (4b reintroduces it.)
3. **`using Stream = PortAudioSharp.Stream;`** at the top of `Program.cs` to disambiguate from `System.IO.Stream` that `ImplicitUsings` pulls in.
4. **`<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`** in `Probe.csproj` for the input/output callbacks' `Buffer.MemoryCopy` over `short*`.
5. **Fire-and-forget output stream lifetime.** `stream.Start()` runs synchronously under try/catch BEFORE any auxiliary cleanup task launches; on `Start()` failure, dispose + rethrow without ever leaking the stream. Test 4a uses a *persistent* output stream (opened once, drained for both the beep and the response audio, stopped on shutdown), so there's no per-cycle cleanup task — the `Start()` failure path is the only one that matters.
6. **Streaming-length cleanup discipline.** Test 3's "scale `done.Wait(timeout)` to recordBuf length" trick does NOT apply because the assistant audio length is unknown a priori. Replaced by the output callback's `_currentChunk == null && Reader.Count == 0 && Reader.Completion.IsCompleted` triple check. The `ManualResetEventSlim doneFlag` carries the signal to the main thread.
7. **`ssh -t` in `bin/probe-cs-4a`** for Ctrl-C propagation from workstation SIGINT.
8. **Always wipe-and-replace the whole publish directory.** A partial `scp Probe` after a rebuild leaves the old `.dll` on the target, which the new launcher loads. `bin/probe-cs-4a` does `ssh "rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a"` BEFORE the `scp -r` (Test 3 pattern).
9. **Startup beep must not be captured by the mic and shipped to OpenAI.** The `_micArmed` flag stays `false` until ~300 ms after the beep is queued to `downstreamChannel` (200 ms beep + 100 ms drain margin). The mic callback discards frames while `!_micArmed`.
New for 4a (none carried forward from prior tests — these are surfaced here for the first time):
10. **`ClientWebSocket` concurrency rule:** one concurrent Send + one concurrent Receive is supported; multiple concurrent Sends (or Receives) is not. Our design has exactly one SendLoop and one ReceiveLoop, so this is safe by construction — but if a future change adds a second sender (e.g. a ping task), it must serialize with the existing one.
11. **Channel completion semantics:** `Writer.Complete()` makes `Reader.Completion.IsCompleted` true ONLY after all written items are consumed. The output callback's triple check accounts for this correctly. Don't change to a different completion sentinel without re-verifying the corner case.
12. **API key file ownership:** the precondition check in `bin/probe-cs-4a` verifies *presence* and *readability* by `pi`, not strict mode 600. The probe itself logs a non-fatal warning if the mode is looser than 600; this lets a slightly looser local setup proceed.
## Verification (hardware only, no automated tests)
One-time setup on the Pi (workstation tells you to do this; `bin/probe-cs-4a` precondition-checks it before publishing):
```sh
sshpass -p 'assistant' ssh pi@192.168.50.115 \
'umask 077 && printf "%s\n" "sk-PASTE-YOUR-KEY-HERE" > ~/.openai_key'
```
Run from workstation:
```sh
./bin/probe-cs-4a
```
Expected console flow (abridged):
```
>> precondition: ~/.openai_key present on Pi. OK.
>> dotnet publish (linux-arm64, self-contained)
>> scp to pi@192.168.50.115:~/probe-cs-4a/ (wipe-and-replace)
>> ssh + run on Pi (Ctrl-C from this terminal stops the probe)
[ws] connecting to wss://api.openai.com/v1/realtime?model=gpt-realtime
[ws] connected
[ws] session.created sess_xxxxxxxxxxxx
[ws] session.updated — session pinned
Using device 3 ('USB Speaker Phone (LISTENAI ...)')
[beep]
Speak now.
[vad] speech started
[vad] speech stopped
[ws] response.created
[audio] first delta received (t = … s since speech_stopped)
[audio] playback drained
[transcript] Hello! How can I help you today?
[ws] closing
bye
```
### Pass criteria
All must hold during a single run on the Pi:
1. **End-to-end round-trip works once.** Probe boots, beeps, you speak a short sentence, the assistant's spoken reply plays back recognisably through the same USB Speaker Phone, the probe exits 0.
2. **Response is intelligible.** The assistant's audio is clear speech, no clipping, no dropouts longer than a single 43 ms callback (one zero-fill is fine and expected on the first chunk arrival).
3. **Round-trip latency.** Time from `[vad] speech stopped` log to `[audio] first delta received` log (i.e. first response audio byte *arriving* at the probe — actual speaker emission adds a fixed ~43 ms output-callback period on top) is **under ~2 s**. Slower is acceptable if explicitly logged; **over ~5 s is a fail.**
4. **No `[status] upstream behind` warnings** during normal operation.
5. **No `[error] WS …` events** during normal operation.
6. **No PortAudio `InputOverflow` status flags** during normal operation.
7. **Workstation Ctrl-C** at any point produces a clean exit: WS closes with normal closure, no stack trace, terminal returns to prompt. Process exit code 0 on Ctrl-C during steady listen; non-zero on Ctrl-C during a failure path is fine.
8. **`response.done`-followed-cleanup signaling works.** Probe exits within ~500 ms of the last audio sample being heard. No long hang.
### What success teaches us for 4b
- Auth + WS handshake + headers + URL are correct.
- `session.update` with `server_vad` works; the server fires `speech_started` / `speech_stopped` correctly on real Pi mic input.
- Base64 decode of `response.audio.delta` produces playable PCM16 at 24 kHz on the USB Speaker Phone (PortAudio plug-resamples 48↔24 at ALSA without trouble).
- The streaming-length cleanup discipline (`_noMoreDeltas` + `Channel.Completion.IsCompleted` + working-chunk check) is correct.
- The `Channel<short[]>` upstream/downstream pattern handles backpressure cleanly.
- `bin/probe-cs-4a`'s `~/.openai_key` precondition pattern is the right key-management story (it'll lift verbatim into 4b and the main assistant).
### What failure would tell us
- **401 / 403 on connect:** API key is wrong or the file's been mangled.
- **WS connects but `session.updated` never arrives:** the session config is malformed for the current Realtime API surface — `modalities`, `turn_detection`, or `voice` is using an outdated field name or value.
- **No `speech_stopped` event ever fires:** `turn_detection` config is wrong for the current API (field renamed?), or the mic is silent (gain too low, wrong device, ALSA plug-rate-conversion broken at 24 kHz). Cross-check by adding a periodic RMS log in the mic callback.
- **`[status] upstream behind` repeats:** UpstreamSendLoop is slower than the mic callback. Most likely `ws.SendAsync` is blocking on TCP. Cheap remediation: bump channel capacity. Real remediation: investigate the WS backpressure.
- **First delta takes >5 s:** server-side issue OR our `instructions` is producing a long thinking pause. Tighten instructions or report and move on.
- **Garbled / chipmunk-speed playback:** sample-rate mismatch. Verify the output stream actually opened at 24 kHz (PortAudio might have silently substituted a default).
- **Click at the start of playback:** zero-fill underrun ran before the first delta. Acceptable for a probe; if it bothers you, add a small pre-roll buffer.
- **Probe hangs on exit:** `Writer.Complete()` was never called (receive loop missed `response.done`), or the output callback's triple check is wrong. Investigate via logging.
## Reference paths
- Spec (this doc): `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`
- Plan (next, to write): `docs/superpowers/plans/2026-06-12-test-4a-openai-realtime.md`
- Code (to be written): `tests/04a-realtime-oneshot-cs/`
- Deploy (to be written): `bin/probe-cs-4a`
- Test 3 reference: `tests/03-full-cycle-cs/Program.cs`, `bin/probe-cs-3`, `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
- API key file location: `/home/pi/.openai_key` on the Pi, mode 600, single line of `sk-…`
- Canonical findings: `findings.md` § "C# probe outcome" through § "C# full-cycle outcome"
- OpenAI Realtime API surface as of this brainstorm:
- WebSocket URL: `wss://api.openai.com/v1/realtime?model=gpt-realtime`
- Headers: `Authorization: Bearer <key>`, `OpenAI-Beta: realtime=v1`
- Audio formats: `pcm16` (default 24 kHz Int16), `g711_ulaw`, `g711_alaw`
- Session config keys used: `modalities`, `instructions`, `voice`, `input_audio_format`, `output_audio_format`, `turn_detection`
- `turn_detection.server_vad` subkeys used: `threshold`, `prefix_padding_ms`, `silence_duration_ms`, `create_response`
- Server events consumed: `session.created`, `session.updated`, `input_audio_buffer.speech_started`, `input_audio_buffer.speech_stopped`, `input_audio_buffer.committed`, `response.created`, `response.audio_transcript.delta`, `response.audio.delta`, `response.audio.done`, `response.done`, `error`
+102
View File
@@ -211,6 +211,108 @@ The Python Test 2 (openwakeword "alexa" listener with beep on detect) was re-imp
- Code: `tests/02-wakeword-cs/` - Code: `tests/02-wakeword-cs/`
- Deploy: `bin/probe-cs-2` - Deploy: `bin/probe-cs-2`
## C# full-cycle outcome (2026-06-12 — Test 3)
Test 3 stitched the C# Test 1 (record + play) and Test 2 (wakeword + beep) into one state machine: idle → "alexa" → beep → record 5 s → playback → idle. Probe lives at `tests/03-full-cycle-cs/`; deploy script `bin/probe-cs-3`. All seven spec pass criteria held on hardware on the first run-after-fix (see "Key gotchas" below).
### What we proved
- **The single-input-stream + state-machine + per-cycle-output-streams shape works on this hardware.** One PortAudio input stream open for the process lifetime feeds a `BlockingCollection<short[]>` (bounded 16); a single consumer thread dispatches each 80 ms frame through IDLE / BEEPING / RECORDING / PLAYBACK. No locks, no `Interlocked``state` is consumer-thread-local; the only cross-thread datum is a `volatile bool` in a `PlaybackDoneFlag`.
- **Detector gating during PLAYBACK prevents self-trigger.** The recorded user audio comes back out the speaker and re-enters the input stream; because the consumer's PLAYBACK branch discards frames (never calls `model.Predict`), the playback never re-triggers detection.
- **`WakewordModel.Reset()` on PLAYBACK→IDLE re-arms the warmup guard correctly.** Five lines (`_rawRingFill = 0; _melRing.Clear(); _embRing.Clear(); _framesSeen = 0`) restore the model to the same state it has just after construction; the existing `WarmupFrames = 16` then suppresses the next ~1.3 s of `Predict` calls until the rings refill with fresh post-playback audio. Next "alexa" after playback ends is detected reliably and quickly (within target).
- **3+ cycles back-to-back, no degradation.** Detection rate holds, CPU stays in IDLE-state range between cycles, no stuck states.
- **No `[status]` or `[error]` warnings during normal operation.** Input doesn't overflow during PLAYBACK (the consumer keeps draining the queue into the discard branch), consumer doesn't fall behind, no fire-and-forget timeouts after the fix below.
- **Ctrl-C from the workstation cleanly exits the probe.** Same `ssh -t` + `Console.CancelKeyPress` + `cts.Cancel()` pattern as Test 2.
### Key gotchas
- **The fire-and-forget cleanup task's `done.Wait` timeout must scale to the buffer length, not be a flat constant.** Test 2's `FireAndForgetBeep` used `done.Wait(TimeSpan.FromSeconds(2))` to guard against silently-faulted PortAudio callbacks (PortAudio swallows callback exceptions). For a 200 ms beep, 2 s is generous; the cleanup task's wait always returns "set", then sleeps 200 ms and disposes. Test 3's `FireAndForgetPlayback` initially inherited the same flat 2 s timeout — but the recording playback runs for ~5 s, so the timeout fired mid-playback, the cleanup task called `stream.Stop()` 200 ms later, and the audio cut at ~2.2 s. **First hardware run produced "[error] recording playback timed out — callback may have faulted" on every cycle.** Fix: compute `expectedSeconds = buf.Length / SampleRate` and use `TimeSpan.FromSeconds(expectedSeconds + 2.0)` as the cleanup wait. Commit `a7d408e`.
- **Partial deploys are silently wrong: a self-contained .NET publish uses `Probe` (native launcher) + `Probe.dll` (managed code).** Copying only `Probe` after a rebuild leaves the old `.dll` on the target, which the new launcher loads — so the binary on the Pi reflects code from the previous build. Always wipe-and-replace the whole publish directory (the pattern already in `bin/probe-cs-3`: `rm -rf ~/probe-cs-3 && mkdir ~/probe-cs-3 && scp -r $PUBLISH_DIR/. pi:~/probe-cs-3/`). Don't do incremental `scp Probe`.
- **`PlaybackDoneFlag` should be created per cycle, not reused.** A theoretical late `Set()` from a delayed cleanup `Task.Run` (cycle N) could land on the shared flag while the consumer is in cycle N+1's PLAYBACK, causing a spurious immediate PLAYBACK→IDLE that skips playback. Allocating a fresh `PlaybackDoneFlag()` inside the RECORDING→PLAYBACK transition (and reading with `playbackDone!.Consume()` in PLAYBACK) sends the stale `Set()` to a now-unreachable object. Commit `070cc07`. Not observed in practice but cheap to defend.
- **Log the cleanup-task's `done.Wait` timeout return.** Without the log, a silently-faulted PortAudio callback produces "PLAYBACK→IDLE with no audio" with no diagnostic — exactly the failure mode the timeout was supposed to defend against. Both `FireAndForgetBeep` and `FireAndForgetPlayback` now log `[error] ... timed out — callback may have faulted` on `!done.Wait(timeout)`. Commit `070cc07`.
### Implications for the main assistant
- The whole shape — persistent input stream, single-consumer state machine, per-cycle output streams, fire-and-forget cleanup with timeout scaled to buffer length — is good to lift directly into the main Pi client. The main assistant adds: Opus encoding on the recorded buffer before sending to the Realtime WebSocket, an Opus decoder feeding the output stream during TTS playback, and replacing the local 5 s recording window with VAD-driven endpointing. None of those replace the state-machine shape; they slot in.
- `WakewordModel` should be promoted out of `tests/` into a real source tree (e.g. `src/Audio/Wakeword/`) when the main assistant starts. Test 3's copy is the live version (it has `Reset()`); Test 2's is frozen.
- The "always wipe-and-replace the whole publish directory" deploy pattern needs to be the default in the main assistant's deploy script too. A partial `scp` of `Probe` is a footgun every time the .dll changes (i.e. every code change).
### Reference paths (C# full-cycle probe)
- Spec: `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`
- Plan: `docs/superpowers/plans/2026-06-12-test-3-full-cycle-csharp.md`
- Code: `tests/03-full-cycle-cs/`
- Deploy: `bin/probe-cs-3`
## Test 4a outcome (2026-06-12 — OpenAI Realtime one-shot voice round-trip in C#)
Test 4a opens a WebSocket to `wss://api.openai.com/v1/realtime?model=gpt-realtime`, plays a startup beep, streams mic audio to the server VAD endpoint, plays the assistant's audio reply back through the same USB Speaker Phone, and exits. Probe lives at `tests/04a-realtime-oneshot-cs/`; deploy script `bin/probe-cs-4a`. All 8 pass criteria met on hardware after a four-round iteration to match the GA Realtime API surface (the spec was written against beta-era docs from openai/openai-realtime-api-beta).
### What we proved
- **`ClientWebSocket` + two background `Task.Run` loops (ReceiveLoop, UpstreamSendLoop) + bounded `Channel<short[]>` on each direction** is the right shape on .NET 9 / linux-arm64 for a Realtime API client. No third-party WebSocket lib needed; `System.Text.Json` is fine for event parsing via `JsonDocument`.
- **One PortAudio input stream + one persistent output stream at 24 kHz mono Int16, both on the USB Speaker Phone, coexist cleanly.** PortAudio plug-resamples 48↔24 at ALSA on this device. No client-side resampling.
- **Server VAD endpointing works out of the box.** `turn_detection: { type: "server_vad", threshold: 0.5, prefix_padding_ms: 300, silence_duration_ms: 500, create_response: true }` (nested under `session.audio.input`) fires `speech_started` / `speech_stopped` correctly on real Pi mic input; server auto-commits the buffer and creates the response.
- **Round-trip latency is excellent.** End-of-speech (server VAD `speech_stopped`) to first response audio sample arriving at the probe: **0.44 s** measured on the test "hello" turn. Spec target was under 2 s; we beat it by 4×.
- **Streaming-length cleanup discipline works.** Output callback signals `doneFlag` when `CurrentChunk == null && Reader.Count == 0 && NoMoreDeltas` (the `NoMoreDeltas` `volatile bool` is set on `response.done` after `Writer.Complete()`; reading it is cheaper than touching `Reader.Completion`). Probe exits ~immediately after `response.done`.
- **`~/.openai_key` precondition pattern in `bin/probe-cs-4a` works.** Pre-publish check via `sshpass ... 'test -f ~/.openai_key && test -r ~/.openai_key'` catches a missing key at the deploy step rather than 30 s into publish + scp. Set with `umask 077 && cat > ~/.openai_key <<KEY` on the Pi.
### Key gotchas — Realtime API GA migration
The spec was written against the older beta API surface (the openai/openai-realtime-api-beta library docs, which were the largest indexed source). The GA API rejected every beta-era piece of our session.update payload, surfacing one error per iteration. The full diff (commit `4cfb0d4`):
1. **No `OpenAI-Beta: realtime=v1` request header.** GA rejects it as `beta_api_shape_disabled`. Only `Authorization: Bearer <key>` is required for GA.
2. **`session.update` payload is nested under `session.audio.{input,output}` instead of flat at the top of `session`.** Beta had `session.input_audio_format`, `session.output_audio_format`, `session.voice`, `session.turn_detection`. GA has `session.audio.input.format`, `session.audio.input.turn_detection`, `session.audio.output.format`, `session.audio.output.voice`. Also `session.type = "realtime"` and `session.model = "gpt-realtime"` go inline at the top of `session`.
3. **`session.audio.{input,output}.format` is now an OBJECT, not a bare string.** Beta accepted `"pcm16"`. GA requires `{ "type": "audio/pcm", "rate": 24000 }`. The error was `Invalid type for 'session.audio.input.format': expected an object, but got a string instead.`
4. **`session.modalities` is no longer accepted.** Server rejected with `Unknown parameter: 'session.modalities'`. Dropped from our payload; server defaults are fine (audio + text output). The GA equivalent for constraining output is likely `output_modalities` (matches Responses API convention) — not needed for the probe.
5. **Server events for response audio are renamed `response.audio.*` → `response.output_audio.*`.** Beta had `response.audio.delta` / `.done` / `response.audio_transcript.delta` / `.done`. GA has `response.output_audio.delta` / `.done` / `response.output_audio_transcript.delta` / `.done`. **This is the silent failure mode**: the receive loop's switch default branch quietly logs `[ws] ignored type=…` instead of writing the decoded audio to the output channel. The probe logs "session pinned", VAD fires, response completes, transcript is empty, no audio is heard. Took one round of hardware verification to surface; trivially diagnosed via the `ignored type=…` log lines.
6. **GA emits a stream of conversation/response framing events** that don't exist in beta: `conversation.item.added` / `.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `rate_limits.updated`. Logged as `[ws] ignored type=…` initially; now silenced in our switch so they don't bury real signal.
The current Realtime model name `gpt-realtime` is still accepted (the dev docs mention `gpt-realtime-2` as well; we didn't need to swap).
### Other gotchas
- **Mid-startup WS error must route through cleanup.** Initial implementation had a single `try { … } finally { … }` around main. When the receive loop processed an error event and called `cts.Cancel()` mid-`SendSessionUpdate`, the `await ws.SendAsync(…, cts.Token)` threw `TaskCanceledException`, which propagated past the finally as an unhandled exception (PortAudio + ws Dispose still ran from finally, but the process exited non-zero with a stack trace). Fix: add `catch (OperationCanceledException) { /* expected on cancel */ } catch (Exception ex) { log; ExitCode = 7 }` between the outer try and finally. Probe now logs the `[error] WS error event:` from the receive loop and exits cleanly. Commit `4cfb0d4`.
- **`Environment.Exit(N)` skips the outer `finally` and leaks PortAudio**, but is OK in the LoadApiKey path because it runs before `PortAudio.Initialize()`. The `Environment.ExitCode = N; return;` pattern in WS-connect / stream-Start failure paths preserves the finally correctly.
- **CA1416 warning on `File.GetUnixFileMode`** is benign; the call is guarded by `try { … } catch (PlatformNotSupportedException) { }`. The build emits one warning consistently; no action needed.
- **ALSA shutdown noise on first run.** Two `alsa_snd_pcm_drop` "failed" lines appeared from PortAudio's cleanup on the *failed* runs (cts.Cancel from an error event tore down both streams mid-init). Did NOT appear on the successful run. Looks like a PortAudio cleanup-on-abort quirk; only present on error paths.
### Pass criteria
| # | Criterion | Result |
|-|-|-|
| 1 | End-to-end round-trip works once | ✅ Probe boots, beeps, user speaks, audible reply, exits 0 |
| 2 | Response intelligible | ✅ Clean speech, no clipping, no dropouts |
| 3 | Latency < 2 s | ✅ **0.44 s** end-of-speech to first delta |
| 4 | No `[status] upstream behind` | ✅ None |
| 5 | No `[error] WS …` during normal flow | ✅ None |
| 6 | No PortAudio `InputOverflow` | ✅ None |
| 7 | Ctrl-C clean exit | ✅ Same cleanup path as the successful run, which exited cleanly |
| 8 | Exits within ~500 ms of last audio sample | ✅ `response.done``[ws] closing``bye` is instant |
### Implications for Test 4b
- **GA API surface is now proven.** Test 4b inherits the migrated `SendSessionUpdate` payload, the renamed receive-loop switch cases, the silenced GA framing events, and the outer try/catch shape. Lift them directly.
- **The "spec written against beta docs" failure mode is a real risk for any future Realtime API work.** When LLM-fetched docs (context7, training data) disagree with the live server, **only the server is authoritative**. Pattern: send a minimal session.update, observe the error response, iterate. Our four-round iteration took ~10 min total.
- **The "[ws] ignored type=…" log line on the default branch is the diagnostic** that surfaced the renames. Keep it (or an equivalent) in 4b's receive loop. Without it, missing audio could have been silently attributed to a thousand other causes.
- **Server VAD parameters (`threshold: 0.5`, `silence_duration_ms: 500`, `prefix_padding_ms: 300`) work well on this hardware** — natural-feeling end-of-turn after ~500 ms of silence, no clipped openings of user speech. Lift unchanged into 4b.
- **`Environment.Exit` vs `Environment.ExitCode = N; return;`** — the latter is the correct pattern for failure paths that need PortAudio + WS cleanup. 4b will have more failure paths (wakeword model init, audio rate conversion, etc.); use the ExitCode pattern consistently.
- **The startup beep pattern (queue beep onto the persistent output channel before `_micArmed = true`)** works — beep does not get captured and shipped to OpenAI. Carry into 4b.
- **The 16↔24 kHz coexistence question is the one major thing 4a did not address.** 4a is 24 kHz end-to-end. 4b must deal with the wakeword model requiring 16 kHz input while OpenAI requires 24 kHz pcm16 — either a second input stream, an in-process resampler, or open one stream at 24 kHz and downsample for the wakeword. The spec/plan for 4b will need to pick.
### Reference paths (Test 4a)
- Spec: `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`
- Plan: `docs/superpowers/plans/2026-06-12-test-4a-openai-realtime.md`
- Code: `tests/04a-realtime-oneshot-cs/`
- Deploy: `bin/probe-cs-4a`
- GA Realtime API headers / URL / session.update shape (as discovered against the live server, 2026-06-12):
- `wss://api.openai.com/v1/realtime?model=gpt-realtime`
- Header `Authorization: Bearer <key>` (NO `OpenAI-Beta`)
- `session.update` payload: `{ type: "session.update", session: { type: "realtime", model: "gpt-realtime", instructions: "...", audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: { type: "server_vad", threshold, prefix_padding_ms, silence_duration_ms, create_response } }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "alloy" } } } }`
- Server events to handle: `session.created`, `session.updated`, `input_audio_buffer.speech_started`, `input_audio_buffer.speech_stopped`, `input_audio_buffer.committed`, `response.created`, `response.output_audio_transcript.delta`, `response.output_audio.delta`, `response.output_audio.done`, `response.output_audio_transcript.done`, `response.done`, `error`
- Server events safe to ignore for a probe: `conversation.item.added`, `conversation.item.done`, `response.output_item.added`, `response.output_item.done`, `response.content_part.added`, `response.content_part.done`, `rate_limits.updated`
## Reference paths ## Reference paths
- Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md` - Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
+15 -1
View File
@@ -59,8 +59,11 @@ internal sealed class WakewordModel : IDisposable
}; };
_mel = new InferenceSession(melPath, opts); _mel = new InferenceSession(melPath, opts);
PrintShapes(Path.GetFileName(melPath), _mel);
_emb = new InferenceSession(embeddingPath, opts); _emb = new InferenceSession(embeddingPath, opts);
PrintShapes(Path.GetFileName(embeddingPath), _emb);
_cls = new InferenceSession(classifierPath, opts); _cls = new InferenceSession(classifierPath, opts);
PrintShapes(Path.GetFileName(classifierPath), _cls);
// Discover input names + assert shape geometry. // Discover input names + assert shape geometry.
_melInputName = _mel.InputMetadata.Keys.Single(); _melInputName = _mel.InputMetadata.Keys.Single();
@@ -70,6 +73,17 @@ internal sealed class WakewordModel : IDisposable
AssertClassifierShape(_cls); // [batch, 16, 96], dtype float AssertClassifierShape(_cls); // [batch, 16, 96], dtype float
} }
private static void PrintShapes(string filename, InferenceSession sess)
{
Console.Error.WriteLine($"[wakeword-model] {filename}");
foreach (var kv in sess.InputMetadata)
Console.Error.WriteLine(
$" input '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
foreach (var kv in sess.OutputMetadata)
Console.Error.WriteLine(
$" output '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
}
private static void AssertEmbeddingShape(InferenceSession sess) private static void AssertEmbeddingShape(InferenceSession sess)
{ {
if (!sess.InputMetadata.TryGetValue(EmbeddingInputName, out var meta)) if (!sess.InputMetadata.TryGetValue(EmbeddingInputName, out var meta))
@@ -144,7 +158,7 @@ internal sealed class WakewordModel : IDisposable
var melOut = melResults.First().AsTensor<float>(); // shape (1, 1, n_frames, 32) var melOut = melResults.First().AsTensor<float>(); // shape (1, 1, n_frames, 32)
// Apply openwakeword's `x / 10 + 2` transform and append each new frame to _melRing. // Apply openwakeword's `x / 10 + 2` transform and append each new frame to _melRing.
// FIXED: actual mel output shape is (1, 1, n_frames, 32) n_frames is at Dimensions[2], NOT [1]. // Mel output shape is (1, 1, n_frames, 32). Note n_frames is at Dimensions[2], not [1].
int nFrames = melOut.Dimensions[2]; int nFrames = melOut.Dimensions[2];
for (int f = 0; f < nFrames; f++) for (int f = 0; f < nFrames; f++)
{ {
+24
View File
@@ -0,0 +1,24 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>FullCycleProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.26.0" />
</ItemGroup>
<ItemGroup>
<Content Include="models/*.onnx">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project>
+387
View File
@@ -0,0 +1,387 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Runtime.InteropServices;
using FullCycleProbe;
using Microsoft.ML.OnnxRuntime;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
const int SampleRate = WakewordModel.SampleRate; // 16 000
const int Channels = 1;
const uint BlockFrames = WakewordModel.FrameSamples; // 1280 (80 ms)
const float Threshold = 0.5f;
const double CooldownSeconds = 1.0;
const double BeepHz = 880.0;
const double BeepSeconds = 0.2;
const int BeepingFrames = 3; // 3 * 80 ms = 240 ms (covers 200 ms beep + 40 ms drain/reverb margin)
const int RecordingFrames = 63; // 63 * 80 ms = 5.04 s (rounded up from 5 s to avoid partial-frame handling)
const int RecordBufSamples = RecordingFrames * (int)BlockFrames; // 80640
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio / onnxruntime (which use getenv()) see the values.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
Environment.SetEnvironmentVariable("ORT_LOGGING_LEVEL", "3");
Libc.setenv("ORT_LOGGING_LEVEL", "3", 1);
// ORT 1.26.0: the env var does NOT suppress early GPU-discovery warnings from device_discovery.cc.
// CreateInstanceWithOptions sets the log level at OrtEnv creation time, before EP discovery runs.
var ortEnvOpts = new EnvironmentCreationOptions { logId = "FullCycleProbe", logLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR };
OrtEnv.CreateInstanceWithOptions(ref ortEnvOpts);
PortAudio.Initialize();
WakewordModel? model = null;
try
{
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
Console.WriteLine("Loading WakewordModel...");
var t0 = DateTime.UtcNow;
string modelsDir = Path.Combine(AppContext.BaseDirectory, "models");
model = new WakewordModel(
Path.Combine(modelsDir, "melspectrogram.onnx"),
Path.Combine(modelsDir, "embedding_model.onnx"),
Path.Combine(modelsDir, "alexa.onnx"));
Console.WriteLine($"Loaded in {(DateTime.UtcNow - t0).TotalSeconds:0.00}s.");
var queue = new BlockingCollection<short[]>(boundedCapacity: 16);
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var inParams = new StreamParameters
{
device = device,
channelCount = Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (frameCount != BlockFrames)
{
Console.Error.WriteLine($"[status] unexpected callback frameCount={frameCount} (want {BlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[BlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
if (!queue.TryAdd(frame, 0))
Console.Error.WriteLine("[status] consumer behind, dropping frame");
return StreamCallbackResult.Continue;
};
using var stream = new Stream(
inParams, null, SampleRate, BlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
stream.Start();
short[] beepBuffer = MakeBeep(BeepHz, BeepSeconds, SampleRate);
Console.WriteLine("Listening. Say 'alexa'. Ctrl-C to exit.");
var sw = Stopwatch.StartNew();
TimeSpan lastTrigger = TimeSpan.FromSeconds(-CooldownSeconds);
// State machine (single-writer: this consumer thread).
State state = State.Idle;
int beepFrames = 0;
int recordOffset = 0;
short[] recordBuf = null!; // assigned on BEEPING→RECORDING transition
PlaybackDoneFlag? playbackDone = null;
try
{
while (!cts.IsCancellationRequested)
{
short[] frame = queue.Take(cts.Token);
switch (state)
{
case State.Idle:
{
float score = model.Predict(frame);
var now = sw.Elapsed;
if (score >= Threshold && (now - lastTrigger).TotalSeconds >= CooldownSeconds)
{
Console.WriteLine($"DETECTED alexa score={score:0.000} t={now.TotalSeconds:0.0}s");
lastTrigger = now;
try
{
FireAndForgetBeep(device, beepBuffer);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] beep stream failed to start: {ex.Message}");
// Stay in IDLE — no cycle to abort yet.
break;
}
Console.WriteLine("[state] BEEPING");
state = State.Beeping;
beepFrames = 0;
}
break;
}
case State.Beeping:
{
beepFrames++;
if (beepFrames >= BeepingFrames)
{
Console.WriteLine("[state] RECORDING");
state = State.Recording;
recordBuf = new short[RecordBufSamples];
recordOffset = 0;
}
break;
}
case State.Recording:
{
unsafe
{
fixed (short* src = &frame[0])
fixed (short* dst = &recordBuf[recordOffset])
Buffer.MemoryCopy(src, dst, BlockFrames * sizeof(short), BlockFrames * sizeof(short));
}
recordOffset += (int)BlockFrames;
if (recordOffset >= RecordBufSamples)
{
try
{
playbackDone = new PlaybackDoneFlag();
FireAndForgetPlayback(device, recordBuf, playbackDone);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] playback stream failed to start: {ex.Message}");
// Drop the recording, reset model, return to IDLE.
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
break;
}
Console.WriteLine("[state] PLAYBACK");
state = State.Playback;
}
break;
}
case State.Playback:
{
// Discard input during playback (gate the detector).
if (playbackDone!.Consume())
{
model.Reset();
Console.WriteLine("[state] IDLE");
state = State.Idle;
}
break;
}
}
}
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
stream.Stop();
Console.WriteLine("\nbye");
}
finally
{
model?.Dispose();
PortAudio.Terminate();
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static void FireAndForgetBeep(int device, short[] beepBuffer)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = beepBuffer.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &beepBuffer[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= beepBuffer.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
Task.Run(() =>
{
// 2 s timeout so a failed callback (which PortAudio silently swallows
// and would leave 'done' unset) eventually releases the stream + handle
// instead of leaking them for the process lifetime.
if (!done.Wait(TimeSpan.FromSeconds(2)))
Console.Error.WriteLine("[error] beep playback timed out — callback may have faulted");
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
});
}
static void FireAndForgetPlayback(int device, short[] recordBuf, PlaybackDoneFlag doneFlag)
{
int offset = 0;
var done = new ManualResetEventSlim(false);
var outParams = new StreamParameters
{
device = device,
channelCount = 1,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback playCb = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int remaining = recordBuf.Length - offset;
int take = (int)Math.Min(frameCount, (uint)remaining);
unsafe
{
short* dst = (short*)output.ToPointer();
if (take > 0)
{
fixed (short* src = &recordBuf[offset])
Buffer.MemoryCopy(src, dst, take * sizeof(short), take * sizeof(short));
offset += take;
}
for (int i = take; i < frameCount; i++) dst[i] = 0;
}
if (offset >= recordBuf.Length)
{
done.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
var stream = new Stream(
null, outParams, WakewordModel.SampleRate, 1024, StreamFlags.NoFlag, playCb, IntPtr.Zero);
try
{
stream.Start();
}
catch
{
stream.Dispose();
done.Dispose();
throw;
}
// Timeout = playback wall-clock + 2 s safety margin. Test 2's beep cleanup uses
// a flat 2 s because beep is 200 ms; for a 5 s recording the same value would
// fire mid-playback, Stop the stream, and cut the audio.
double expectedSeconds = recordBuf.Length / (double)WakewordModel.SampleRate;
var waitTimeout = TimeSpan.FromSeconds(expectedSeconds + 2.0);
Task.Run(() =>
{
if (!done.Wait(waitTimeout))
Console.Error.WriteLine("[error] recording playback timed out — callback may have faulted");
Thread.Sleep(200);
stream.Stop();
stream.Dispose();
done.Dispose();
doneFlag.Set(); // tell the consumer it's safe to flip PLAYBACK→IDLE
});
}
enum State { Idle, Beeping, Recording, Playback }
// Volatile flag set by the playback cleanup task and consumed by the consumer thread.
// Wrapped in a tiny class so we can pass it by reference into the helper.
sealed class PlaybackDoneFlag
{
private volatile bool _set;
public void Set() => _set = true;
public bool Consume()
{
if (!_set) return false;
_set = false;
return true;
}
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}
+226
View File
@@ -0,0 +1,226 @@
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
namespace FullCycleProbe;
// Streaming wakeword inference port of openwakeword 0.6.0's predict pipeline.
// References (read alongside this file):
// openwakeword/utils.py:AudioFeatures._streaming_features (mel + embedding stages)
// openwakeword/utils.py:AudioFeatures._streaming_melspectrogram
// openwakeword/utils.py:AudioFeatures._get_embeddings
// openwakeword/model.py:Model.predict (classifier stage)
internal sealed class WakewordModel : IDisposable
{
// Audio I/O geometry — per-call input contract.
public const int FrameSamples = 1280; // 80 ms @ 16 kHz; each Predict() call.
public const int SampleRate = 16_000;
// Mel model: input is last (FrameSamples + MelContextSamples) raw samples
// -- the +480 is openwakeword's `-n_samples-160*3:` slice (3 hops of context).
public const int MelContextSamples = 480; // 160 * 3
public const int MelInputSamples = FrameSamples + MelContextSamples; // 1760
public const int MelBins = 32; // melspectrogram model output dim
public const int MelBufferMaxFrames = 970; // 10 * 97 (openwakeword `melspectrogram_max_len`)
// Embedding model: 76-mel-frame window in, 96-d embedding out.
public const int EmbeddingWindowMelFrames = 76;
public const int EmbeddingDim = 96;
public const int EmbeddingBufferMax = 120; // openwakeword `feature_buffer_max_len`
public const string EmbeddingInputName = "input_1"; // openwakeword convention; assert at startup
// Classifier: 16 embeddings in, scalar score out.
public const int ClassifierEmbeddings = 16;
// Skip the first N Predict() calls — buffer fill-up window.
public const int WarmupFrames = ClassifierEmbeddings; // 16 frames ≈ 1.28 s
private readonly InferenceSession _mel;
private readonly InferenceSession _emb;
private readonly InferenceSession _cls;
private readonly string _melInputName; // discovered at startup
private readonly string _clsInputName; // discovered at startup
private readonly short[] _rawRing = new short[MelInputSamples];
private int _rawRingFill = 0; // samples buffered (≤ MelInputSamples)
private readonly List<float[]> _melRing = new(MelBufferMaxFrames); // each entry is a length-32 mel frame
private readonly List<float[]> _embRing = new(EmbeddingBufferMax); // each entry is a length-96 embedding
private int _framesSeen = 0;
public WakewordModel(string melPath, string embeddingPath, string classifierPath)
{
var opts = new SessionOptions
{
IntraOpNumThreads = 1,
InterOpNumThreads = 1,
LogSeverityLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_ERROR,
};
_mel = new InferenceSession(melPath, opts);
PrintShapes(Path.GetFileName(melPath), _mel);
_emb = new InferenceSession(embeddingPath, opts);
PrintShapes(Path.GetFileName(embeddingPath), _emb);
_cls = new InferenceSession(classifierPath, opts);
PrintShapes(Path.GetFileName(classifierPath), _cls);
// Discover input names + assert shape geometry.
_melInputName = _mel.InputMetadata.Keys.Single();
AssertEmbeddingShape(_emb); // input_1: [batch, 76, 32, 1], dtype float
_clsInputName = _cls.InputMetadata.Keys.Single();
AssertClassifierShape(_cls); // [batch, 16, 96], dtype float
}
private static void PrintShapes(string filename, InferenceSession sess)
{
Console.Error.WriteLine($"[wakeword-model] {filename}");
foreach (var kv in sess.InputMetadata)
Console.Error.WriteLine(
$" input '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
foreach (var kv in sess.OutputMetadata)
Console.Error.WriteLine(
$" output '{kv.Key}': shape=[{string.Join(",", kv.Value.Dimensions)}] dtype={kv.Value.ElementType.Name}");
}
private static void AssertEmbeddingShape(InferenceSession sess)
{
if (!sess.InputMetadata.TryGetValue(EmbeddingInputName, out var meta))
throw new InvalidOperationException(
$"embedding_model.onnx: expected input named '{EmbeddingInputName}', got [{string.Join(",", sess.InputMetadata.Keys)}]");
var d = meta.Dimensions;
// Expected: [batch, 76, 32, 1] — batch may be -1 (dynamic).
if (d.Length != 4 || d[1] != EmbeddingWindowMelFrames || d[2] != MelBins || d[3] != 1)
throw new InvalidOperationException(
$"embedding_model.onnx: expected input shape [batch,{EmbeddingWindowMelFrames},{MelBins},1], got [{string.Join(",", d)}]");
if (meta.ElementType != typeof(float))
throw new InvalidOperationException($"embedding_model.onnx: expected Single input, got {meta.ElementType.Name}");
}
private static void AssertClassifierShape(InferenceSession sess)
{
var inputName = sess.InputMetadata.Keys.Single();
var meta = sess.InputMetadata[inputName];
var d = meta.Dimensions;
// Expected: [batch, 16, 96] — batch may be -1.
if (d.Length != 3 || d[1] != ClassifierEmbeddings || d[2] != EmbeddingDim)
throw new InvalidOperationException(
$"alexa.onnx: expected input shape [batch,{ClassifierEmbeddings},{EmbeddingDim}], got [{string.Join(",", d)}]");
if (meta.ElementType != typeof(float))
throw new InvalidOperationException($"alexa.onnx: expected Single input, got {meta.ElementType.Name}");
}
public float Predict(short[] frame1280)
{
if (frame1280.Length != FrameSamples)
throw new ArgumentException($"Expected {FrameSamples} samples, got {frame1280.Length}");
// 1. Append 1280 new samples to the raw ring (shift older samples down if full).
if (_rawRingFill < MelInputSamples)
{
int copyToFront = Math.Min(MelInputSamples - _rawRingFill, FrameSamples);
Array.Copy(frame1280, 0, _rawRing, _rawRingFill, copyToFront);
_rawRingFill += copyToFront;
if (copyToFront < FrameSamples)
{
// Boundary case: ring was partially full and the new frame overshoots
// remaining capacity. Fires exactly once during warm-up (typically call 2,
// when _rawRingFill = 1280 and the incoming 1280 samples overshoot the
// remaining 480 capacity). Shift the older samples left to make room,
// then write the leftover at the tail.
int leftover = FrameSamples - copyToFront;
Array.Copy(_rawRing, leftover, _rawRing, 0, MelInputSamples - leftover);
Array.Copy(frame1280, copyToFront, _rawRing, MelInputSamples - leftover, leftover);
}
}
else
{
// Shift older samples left by FrameSamples, then append new at the tail.
Array.Copy(_rawRing, FrameSamples, _rawRing, 0, MelInputSamples - FrameSamples);
Array.Copy(frame1280, 0, _rawRing, MelInputSamples - FrameSamples, FrameSamples);
}
// Skip everything until we have the full mel-context window primed.
if (_rawRingFill < MelInputSamples)
{
_framesSeen++;
return 0f;
}
// 2. Mel stage: feed the full _rawRing as float32 (1, MelInputSamples) into mel model.
var melInputData = new float[MelInputSamples];
for (int i = 0; i < MelInputSamples; i++) melInputData[i] = _rawRing[i]; // int16 → float32, NO normalisation
var melInputTensor = new DenseTensor<float>(melInputData, new[] { 1, MelInputSamples });
using var melResults = _mel.Run(new[] {
NamedOnnxValue.CreateFromTensor(_melInputName, melInputTensor)
});
var melOut = melResults.First().AsTensor<float>(); // shape (1, 1, n_frames, 32)
// Apply openwakeword's `x / 10 + 2` transform and append each new frame to _melRing.
// Mel output shape is (1, 1, n_frames, 32). Note n_frames is at Dimensions[2], not [1].
int nFrames = melOut.Dimensions[2];
for (int f = 0; f < nFrames; f++)
{
var bin = new float[MelBins];
for (int b = 0; b < MelBins; b++)
bin[b] = melOut[0, 0, f, b] / 10f + 2f;
_melRing.Add(bin);
}
if (_melRing.Count > MelBufferMaxFrames)
_melRing.RemoveRange(0, _melRing.Count - MelBufferMaxFrames);
// 3. Embedding stage: need ≥ 76 mel frames; slice the last 76 → (1, 76, 32, 1).
if (_melRing.Count < EmbeddingWindowMelFrames)
{
_framesSeen++;
return 0f;
}
var embInputData = new float[EmbeddingWindowMelFrames * MelBins];
int startMel = _melRing.Count - EmbeddingWindowMelFrames;
for (int f = 0; f < EmbeddingWindowMelFrames; f++)
Array.Copy(_melRing[startMel + f], 0, embInputData, f * MelBins, MelBins);
var embInputTensor = new DenseTensor<float>(embInputData, new[] { 1, EmbeddingWindowMelFrames, MelBins, 1 });
using var embResults = _emb.Run(new[] {
NamedOnnxValue.CreateFromTensor(EmbeddingInputName, embInputTensor)
});
var embOut = embResults.First().AsTensor<float>(); // shape (1, 1, 1, 96)
var newEmb = new float[EmbeddingDim];
for (int i = 0; i < EmbeddingDim; i++) newEmb[i] = embOut[0, 0, 0, i];
_embRing.Add(newEmb);
if (_embRing.Count > EmbeddingBufferMax)
_embRing.RemoveRange(0, _embRing.Count - EmbeddingBufferMax);
_framesSeen++;
// 4. Warm-up + classifier stage.
if (_embRing.Count < ClassifierEmbeddings || _framesSeen <= WarmupFrames)
return 0f;
var clsInputData = new float[ClassifierEmbeddings * EmbeddingDim];
int startEmb = _embRing.Count - ClassifierEmbeddings;
for (int i = 0; i < ClassifierEmbeddings; i++)
Array.Copy(_embRing[startEmb + i], 0, clsInputData, i * EmbeddingDim, EmbeddingDim);
var clsInputTensor = new DenseTensor<float>(clsInputData, new[] { 1, ClassifierEmbeddings, EmbeddingDim });
using var clsResults = _cls.Run(new[] {
NamedOnnxValue.CreateFromTensor(_clsInputName, clsInputTensor)
});
var clsOut = clsResults.First().AsTensor<float>(); // shape (1, 1)
return clsOut[0, 0];
}
public void Reset()
{
_rawRingFill = 0;
_melRing.Clear();
_embRing.Clear();
_framesSeen = 0;
}
public void Dispose()
{
_mel.Dispose();
_emb.Dispose();
_cls.Dispose();
}
}
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,18 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>RealtimeOneShotProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
</ItemGroup>
</Project>
+542
View File
@@ -0,0 +1,542 @@
using System.Net.WebSockets;
using System.Runtime.InteropServices;
using System.Text;
using System.Text.Json;
using System.Threading.Channels;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio (which uses getenv()) sees the value.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
string apiKey = LoadApiKey();
PortAudio.Initialize();
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var upstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.UpstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = true,
SingleReader = true,
});
var downstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.DownstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = false,
SingleReader = true,
});
var doneFlag = new ManualResetEventSlim(false);
var state = new SharedState();
var ws = new ClientWebSocket();
ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
// No OpenAI-Beta header — the GA Realtime API rejects beta-shaped requests with
// "beta_api_shape_disabled". Documented at developers.openai.com/api/docs/guides/realtime-websocket.
Stream? inputStream = null;
Stream? outputStream = null;
Task? receiveTask = null;
Task? sendTask = null;
try
{
var uri = new Uri($"wss://api.openai.com/v1/realtime?model={C.ModelName}");
Console.WriteLine($"[ws] connecting to {uri}");
try
{
await ws.ConnectAsync(uri, cts.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] WS connect failed: {ex.Message}");
if (ex.InnerException is HttpRequestException httpEx)
Console.Error.WriteLine($"[error] inner: HTTP {(int?)httpEx.StatusCode}");
Environment.ExitCode = 3;
return;
}
Console.WriteLine("[ws] connected");
receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel, cts, state));
await SendSessionUpdate(ws, cts.Token);
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
inputStream = OpenInputStream(device, upstreamChannel, state);
try { inputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] input stream start failed: {ex.Message}");
Environment.ExitCode = 5;
return;
}
outputStream = OpenOutputStream(device, downstreamChannel, state, doneFlag);
try { outputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] output stream start failed: {ex.Message}");
Environment.ExitCode = 6;
return;
}
sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel, cts, state));
// Startup beep — queued onto downstreamChannel before the receive loop produces any audio.
short[] beep = MakeBeep(C.BeepHz, C.BeepSeconds, C.SampleRate);
Console.WriteLine("[beep]");
await downstreamChannel.Writer.WriteAsync(beep, cts.Token);
// Wait for beep to drain (200 ms beep + 100 ms margin) so it doesn't get captured
// by the mic and shipped to OpenAI.
Thread.Sleep((int)(C.BeepSeconds * 1000) + C.BeepDrainMs);
state.MicArmed = true;
Console.WriteLine("Speak now.");
try
{
doneFlag.Wait(cts.Token);
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
Console.WriteLine("[ws] closing");
}
catch (OperationCanceledException)
{
// Mid-startup cancellation (typically a WS error event fired cts.Cancel while
// the main thread was awaiting SendSessionUpdate / a channel write).
// The receive loop has already logged the underlying [error]; we just route through cleanup.
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] unexpected: {ex.GetType().Name}: {ex.Message}");
Environment.ExitCode = 7;
}
finally
{
// Cancel any background work first so receive/send loops exit cleanly.
cts.Cancel();
if (ws.State == WebSocketState.Open || ws.State == WebSocketState.CloseReceived)
{
try
{
using var closeTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(2));
await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", closeTimeout.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[ws] close failed: {ex.Message}");
}
}
try { inputStream?.Stop(); } catch { }
try { outputStream?.Stop(); } catch { }
try { inputStream?.Dispose(); } catch { }
try { outputStream?.Dispose(); } catch { }
try { ws.Dispose(); } catch { }
try { PortAudio.Terminate(); } catch { }
Console.WriteLine("bye");
}
static string LoadApiKey()
{
string home = Environment.GetEnvironmentVariable("HOME") ?? "/root";
string path = Path.Combine(home, ".openai_key");
if (!File.Exists(path))
{
Console.Error.WriteLine($"[error] API key file not found at {path}");
Environment.Exit(2);
}
// Non-fatal mode-600 check.
try
{
var mode = File.GetUnixFileMode(path);
var leaky = mode & (UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute
| UnixFileMode.OtherRead | UnixFileMode.OtherWrite | UnixFileMode.OtherExecute);
if (leaky != UnixFileMode.None)
Console.Error.WriteLine($"[warn] {path} mode includes group/other perms ({mode}); expected 600");
}
catch (PlatformNotSupportedException) { /* not Unix */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[warn] could not check {path} mode: {ex.Message}");
}
string key = File.ReadAllText(path).Trim();
if (string.IsNullOrEmpty(key))
{
Console.Error.WriteLine($"[error] API key file empty");
Environment.Exit(2);
}
if (!key.StartsWith("sk-"))
{
Console.Error.WriteLine($"[error] API key doesn't start with sk-");
Environment.Exit(2);
}
return key;
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static async Task SendSessionUpdate(ClientWebSocket ws, CancellationToken ct)
{
// GA Realtime API shape: nested session.audio.{input,output} with format + voice + turn_detection
// moved under audio.input. session.type = "realtime" and session.model inline.
// (Beta-era flat shape with session.input_audio_format/turn_detection at the top is rejected.)
var msg = new
{
type = "session.update",
session = new
{
type = "realtime",
model = C.ModelName,
instructions = C.Instructions,
// No 'modalities' field — the beta name. GA defaults are fine; if we ever need to
// constrain to audio-only we'd use 'output_modalities' (the documented GA name in
// newer OpenAI APIs).
audio = new
{
input = new
{
// GA shape: format is now an object with a "type" field, not a bare string.
format = new { type = "audio/pcm", rate = 24_000 },
turn_detection = new
{
type = "server_vad",
threshold = 0.5,
prefix_padding_ms = 300,
silence_duration_ms = 500,
create_response = true,
},
},
output = new
{
format = new { type = "audio/pcm", rate = 24_000 },
voice = C.Voice,
},
},
},
};
byte[] bytes = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(bytes), WebSocketMessageType.Text, endOfMessage: true, ct);
}
static Stream OpenInputStream(int device, Channel<short[]> upstream, SharedState state)
{
var inParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (!state.MicArmed || state.StopSending)
return StreamCallbackResult.Continue;
if (frameCount != C.MicBlockFrames)
{
Console.Error.WriteLine($"[status] unexpected input frameCount={frameCount} (want {C.MicBlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[C.MicBlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, C.MicBlockFrames * sizeof(short), C.MicBlockFrames * sizeof(short));
}
if (!upstream.Writer.TryWrite(frame))
Console.Error.WriteLine("[status] upstream behind, dropping mic frame");
return StreamCallbackResult.Continue;
};
return new Stream(inParams, null, C.SampleRate, C.MicBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static Stream OpenOutputStream(int device, Channel<short[]> downstream, SharedState state, ManualResetEventSlim doneFlag)
{
var outParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int fill = 0;
while (fill < frameCount)
{
// Pull next chunk if we have no working one.
if (state.CurrentChunk == null)
{
if (!downstream.Reader.TryRead(out var nextChunk))
break; // ring empty
state.CurrentChunk = nextChunk;
state.ChunkOffset = 0;
}
// Local non-null alias so the compiler doesn't warn on dereference inside `fixed`.
short[] chunk = state.CurrentChunk!;
int slack = (int)frameCount - fill;
int remaining = chunk.Length - state.ChunkOffset;
int take = Math.Min(remaining, slack);
unsafe
{
short* dst = (short*)output.ToPointer();
fixed (short* src = &chunk[state.ChunkOffset])
Buffer.MemoryCopy(src, dst + fill, take * sizeof(short), take * sizeof(short));
}
state.ChunkOffset += take;
fill += take;
if (state.ChunkOffset >= chunk.Length)
state.CurrentChunk = null;
}
unsafe
{
short* dst = (short*)output.ToPointer();
for (int i = fill; i < (int)frameCount; i++) dst[i] = 0;
}
// Completion = working chunk drained AND channel empty AND no more deltas expected.
// _noMoreDeltas is the spec's "redundant fast path" — set immediately on response.done,
// avoids touching downstream.Reader.Completion (a Task allocation per callback).
if (state.CurrentChunk == null
&& downstream.Reader.Count == 0
&& state.NoMoreDeltas)
{
doneFlag.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
return new Stream(null, outParams, C.SampleRate, C.OutBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static async Task UpstreamSendLoop(ClientWebSocket ws, Channel<short[]> upstream, CancellationTokenSource cts, SharedState state)
{
try
{
await foreach (var frame in upstream.Reader.ReadAllAsync(cts.Token))
{
if (state.StopSending)
{
// After server VAD said the user is done, drop in-flight mic frames and exit
// once the channel is drained. Matches the spec's "_stopSending && empty" exit.
if (upstream.Reader.Count == 0) break;
continue;
}
ReadOnlySpan<byte> bytes = MemoryMarshal.AsBytes(frame.AsSpan());
string base64 = Convert.ToBase64String(bytes);
var msg = new { type = "input_audio_buffer.append", audio = base64 };
byte[] json = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(json), WebSocketMessageType.Text, endOfMessage: true, cts.Token);
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] upstream send: {ex.Message}");
cts.Cancel();
}
}
static async Task ReceiveLoop(ClientWebSocket ws, Channel<short[]> downstream, CancellationTokenSource cts, SharedState state)
{
var transcript = new StringBuilder();
var buffer = new byte[4096];
bool firstDeltaSeen = false;
DateTime speechStoppedTime = DateTime.MinValue;
using var msgBuffer = new MemoryStream();
try
{
while (!cts.IsCancellationRequested)
{
msgBuffer.SetLength(0);
WebSocketReceiveResult result;
do
{
result = await ws.ReceiveAsync(new ArraySegment<byte>(buffer), cts.Token);
if (result.MessageType == WebSocketMessageType.Close) return;
msgBuffer.Write(buffer, 0, result.Count);
} while (!result.EndOfMessage);
msgBuffer.Position = 0;
using var doc = JsonDocument.Parse(msgBuffer);
string type = doc.RootElement.GetProperty("type").GetString() ?? "";
switch (type)
{
case "session.created":
{
string? sid = doc.RootElement.TryGetProperty("session", out var sess)
&& sess.TryGetProperty("id", out var idEl) ? idEl.GetString() : null;
Console.WriteLine($"[ws] session.created {sid}");
break;
}
case "session.updated":
Console.WriteLine("[ws] session.updated — session pinned");
break;
case "input_audio_buffer.speech_started":
Console.WriteLine("[vad] speech started");
break;
case "input_audio_buffer.speech_stopped":
Console.WriteLine("[vad] speech stopped");
speechStoppedTime = DateTime.UtcNow;
state.StopSending = true;
break;
case "input_audio_buffer.committed":
Console.WriteLine("[ws] input_audio_buffer.committed");
break;
case "response.created":
Console.WriteLine("[ws] response.created");
break;
// GA event names are response.output_audio* (beta was response.audio*).
case "response.output_audio_transcript.delta":
if (doc.RootElement.TryGetProperty("delta", out var tEl))
transcript.Append(tEl.GetString());
break;
case "response.output_audio.delta":
{
string b64 = doc.RootElement.GetProperty("delta").GetString() ?? "";
byte[] audioBytes = Convert.FromBase64String(b64);
short[] samples = new short[audioBytes.Length / 2];
Buffer.BlockCopy(audioBytes, 0, samples, 0, audioBytes.Length);
if (!firstDeltaSeen)
{
firstDeltaSeen = true;
double elapsed = speechStoppedTime == DateTime.MinValue
? -1
: (DateTime.UtcNow - speechStoppedTime).TotalSeconds;
Console.WriteLine($"[audio] first delta received (t = {elapsed:0.00} s since speech_stopped)");
}
await downstream.Writer.WriteAsync(samples, cts.Token);
break;
}
case "response.output_audio.done":
Console.WriteLine("[ws] response.output_audio.done");
break;
case "response.output_audio_transcript.done":
// transcript already accumulated; nothing to do here, server signals end of stream.
break;
// GA emits a stream of conversation/response framing events we don't need for the
// probe — quiet them so they don't bury real signal in the log.
case "conversation.item.added":
case "conversation.item.done":
case "response.output_item.added":
case "response.output_item.done":
case "response.content_part.added":
case "response.content_part.done":
case "rate_limits.updated":
break;
case "response.done":
state.NoMoreDeltas = true;
downstream.Writer.Complete();
Console.WriteLine("[ws] response.done");
Console.WriteLine($"[transcript] {transcript}");
break;
case "error":
Console.Error.WriteLine($"[error] WS error event: {doc.RootElement.GetRawText()}");
cts.Cancel();
break;
default:
Console.Error.WriteLine($"[ws] ignored type={type}");
break;
}
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] receive loop: {ex.Message}");
cts.Cancel();
}
}
sealed class SharedState
{
public volatile bool MicArmed;
public volatile bool StopSending;
public volatile bool NoMoreDeltas;
// Output-callback-thread-local; do NOT touch from anywhere else.
public short[]? CurrentChunk;
public int ChunkOffset;
}
static class C
{
public const int SampleRate = 24_000;
public const int Channels = 1;
public const uint MicBlockFrames = 1920; // 40 ms @ 24 kHz
public const uint OutBlockFrames = 1024; // ~43 ms @ 24 kHz
public const string ModelName = "gpt-realtime";
public const string Voice = "alloy";
public const string Instructions = "Reply in one short sentence.";
public const double BeepHz = 880.0;
public const double BeepSeconds = 0.2;
public const int BeepDrainMs = 100;
public const int UpstreamChannelCapacity = 64;
public const int DownstreamChannelCapacity = 64;
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}