Compare commits

...

8 Commits

Author SHA1 Message Date
tes cd07b52f8f Findings: Test 4a OpenAI Realtime one-shot voice round-trip outcome 2026-06-12 19:42:35 +00:00
tes 4cfb0d4b9a Test 4a: migrate session.update + event names to GA Realtime API
Hardware verification surfaced four shape changes from the beta API that
the spec/plan was written against:

1. Remove the OpenAI-Beta: realtime=v1 header (GA rejects it with
   beta_api_shape_disabled).
2. session.update payload nests under session.audio.{input,output} instead
   of flat fields. session.type = "realtime" + session.model are inline.
3. session.audio.{input,output}.format is now an object
   ({ type: "audio/pcm", rate: 24000 }), not a bare "pcm16" string.
4. 'session.modalities' is no longer accepted; dropped (server uses defaults).
5. response.audio.*  →  response.output_audio.*  (delta, done, transcript).

Also silenced a bunch of GA framing events (conversation.item.added /
.done, response.output_item.added / .done, response.content_part.added /
.done, rate_limits.updated) that were burying signal in the [ws] ignored
log lines.

Plus added an outer try/catch around main so a mid-startup WS error event
(which fires cts.Cancel via the receive loop) routes through cleanup
instead of crashing on the await SendSessionUpdate.

Verified on hardware: full round-trip works, t=0.44 s end-of-speech to
first audio sample arriving, assistant reply audible, clean exit.
2026-06-12 19:41:08 +00:00
tes ac8a3f012c Test 4a: bin/probe-cs-4a deploy script with ~/.openai_key precondition 2026-06-12 19:16:32 +00:00
tes a590c96d07 Test 4a: wire NoMoreDeltas + add UpstreamSendLoop stop-and-empty exit
Code-review feedback from Task 2:
- Use state.NoMoreDeltas in the output callback completion check instead
  of downstream.Reader.Completion.IsCompleted (spec called for it as the
  redundant fast path; it was dead code before).
- UpstreamSendLoop now break's when StopSending && upstream channel empty,
  matching the spec's stated exit shape.
2026-06-12 19:15:29 +00:00
tes a1a3b5c981 Test 4a: WS plumbing + audio streams + server VAD round-trip 2026-06-12 19:09:26 +00:00
tes 43b3a1a0f1 Test 4a scaffold: csproj for tests/04a-realtime-oneshot-cs 2026-06-12 19:00:57 +00:00
tes e96b6ffd95 Test 4a: implementation plan
Five tasks: scaffold csproj → write full Program.cs (WS + audio
streams + channels + server VAD round-trip) → bin/probe-cs-4a with
~/.openai_key precondition → one-time Pi key setup → hardware
verification against the spec's 8 pass criteria.
2026-06-12 17:34:25 +00:00
tes dea126f994 Test 4a: spec — OpenAI Realtime API one-shot voice round-trip
Boot → beep → user speaks → server VAD ends turn → assistant audio
plays back → exit 0. No wakeword, no loop, no state machine.

Splits Test 4 into 4a (this spec) and 4b (wakeword + loop + 16↔24 kHz
coexistence, future). 4a debugs the OpenAI Realtime surface — auth, WS
handshake, session.update payload, server VAD, base64 audio, streaming-
length output-stream cleanup — in isolation from 4b's harder concurrency.
2026-06-12 16:59:10 +00:00
6 changed files with 2036 additions and 0 deletions
+36
View File
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-4a-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> precondition: ~/.openai_key present on Pi"
if ! sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'test -f ~/.openai_key && test -r ~/.openai_key'; then
echo "ERROR: ~/.openai_key missing or unreadable on Pi."
echo "Create it on the Pi with:"
echo " sshpass -p '$PI_PASS' ssh $PI_USER@$PI_HOST \\"
echo " 'umask 077 && printf \"%s\\n\" \"sk-PASTE-YOUR-KEY\" > ~/.openai_key'"
exit 1
fi
echo " OK."
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/04a-realtime-oneshot-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-4a/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-4a/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-4a && chmod +x Probe && ./Probe'
@@ -0,0 +1,897 @@
# Test 4a — OpenAI Realtime API one-shot voice round-trip (C#) Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a single-binary throwaway C# probe at `tests/04a-realtime-oneshot-cs/` that has one round-trip spoken conversation with the OpenAI Realtime API: boot → beep → user speaks one sentence → server VAD ends the turn → assistant audio reply plays back through the USB Speaker Phone → exit 0.
**Architecture:** One persistent `ClientWebSocket` to `wss://api.openai.com/v1/realtime?model=gpt-realtime`. Two `Task.Run` loops: ReceiveLoop parses server events with `JsonDocument` and writes decoded audio chunks to a `Channel<short[]>`; UpstreamSendLoop drains a separate `Channel<short[]>` of mic frames and sends them as `input_audio_buffer.append`. PortAudio input stream (24 kHz mono Int16, 1920 frames/cb) writes to the upstream channel via callback; output stream (24 kHz, 1024 frames/cb) drains the downstream channel into the speaker, zero-filling underruns. Completion (`response.done` received AND downstream channel drained AND working chunk empty) is signaled via `ManualResetEventSlim` to the main thread, which closes the WS and exits.
**Tech Stack:** .NET 9 (self-contained linux-arm64), PortAudioSharp2 1.0.6, `System.Net.WebSockets.ClientWebSocket`, `System.Text.Json`, `System.Threading.Channels`. Deploy via `dotnet publish` + `scp` + `ssh -t` to Pi at `pi@192.168.50.115`. No onnxruntime — no wakeword in 4a.
**Spec:** `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`. Read it before starting.
**Branch:** `fresh-start` (already current; do NOT create a new branch).
---
### Task 1: Scaffold `tests/04a-realtime-oneshot-cs/` project skeleton
**Files:**
- Create: `tests/04a-realtime-oneshot-cs/Probe.csproj`
- [ ] **Step 1: Create the project directory**
```bash
mkdir -p tests/04a-realtime-oneshot-cs
```
- [ ] **Step 2: Write `Probe.csproj`**
Create `tests/04a-realtime-oneshot-cs/Probe.csproj` with this exact content. Note the differences from Test 3's csproj: `RootNamespace=RealtimeOneShotProbe`, no `Microsoft.ML.OnnxRuntime` reference (no wakeword in 4a), no `models/*.onnx` content block.
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>RealtimeOneShotProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
</ItemGroup>
</Project>
```
`ClientWebSocket` (from `System.Net.WebSockets`), `JsonDocument` / `JsonSerializer` (from `System.Text.Json`), and `Channel<T>` (from `System.Threading.Channels`) all come from the BCL — no PackageReference needed.
- [ ] **Step 3: Commit the scaffold**
```bash
git add tests/04a-realtime-oneshot-cs/Probe.csproj
git commit -m "Test 4a scaffold: csproj for tests/04a-realtime-oneshot-cs"
```
---
### Task 2: Write `Program.cs` in full
**Files:**
- Create: `tests/04a-realtime-oneshot-cs/Program.cs`
The whole file is shown below. It is self-contained — no helper files, no copied modules. The structure follows Test 3's `Program.cs` shape (top-level statements; helpers, classes, and `static class Libc` at the bottom of the file) with these notable differences:
- No onnxruntime, no `WakewordModel`, no models directory.
- A `ClientWebSocket` + two background `Task.Run` loops (ReceiveLoop, UpstreamSendLoop) replace the consumer-thread state machine.
- The input stream callback's job is to push mic frames to a `Channel<short[]>` (not to call `Predict`).
- The output stream callback drains a different `Channel<short[]>`, zero-fills underruns, and signals completion via a `ManualResetEventSlim` (replacing Test 3's per-cycle `PlaybackDoneFlag`).
- A `SharedState` class holds the three `volatile bool` flags (`MicArmed`, `StopSending`, `NoMoreDeltas`) plus the output callback's per-call drain state (`CurrentChunk`, `ChunkOffset`).
- `LoadApiKey()` reads `~/.openai_key` with mode-600 validation.
- [ ] **Step 1: Write `Program.cs` in full**
Create `tests/04a-realtime-oneshot-cs/Program.cs` with exactly this content:
```csharp
using System.Net.WebSockets;
using System.Runtime.InteropServices;
using System.Text;
using System.Text.Json;
using System.Threading.Channels;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio (which uses getenv()) sees the value.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
string apiKey = LoadApiKey();
PortAudio.Initialize();
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var upstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.UpstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = true,
SingleReader = true,
});
var downstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.DownstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = false,
SingleReader = true,
});
var doneFlag = new ManualResetEventSlim(false);
var state = new SharedState();
var ws = new ClientWebSocket();
ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
ws.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");
Stream? inputStream = null;
Stream? outputStream = null;
Task? receiveTask = null;
Task? sendTask = null;
try
{
var uri = new Uri($"wss://api.openai.com/v1/realtime?model={C.ModelName}");
Console.WriteLine($"[ws] connecting to {uri}");
try
{
await ws.ConnectAsync(uri, cts.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] WS connect failed: {ex.Message}");
if (ex.InnerException is HttpRequestException httpEx)
Console.Error.WriteLine($"[error] inner: HTTP {(int?)httpEx.StatusCode}");
Environment.ExitCode = 3;
return;
}
Console.WriteLine("[ws] connected");
receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel, cts, state));
await SendSessionUpdate(ws, cts.Token);
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
inputStream = OpenInputStream(device, upstreamChannel, state);
try { inputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] input stream start failed: {ex.Message}");
Environment.ExitCode = 5;
return;
}
outputStream = OpenOutputStream(device, downstreamChannel, state, doneFlag);
try { outputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] output stream start failed: {ex.Message}");
Environment.ExitCode = 6;
return;
}
sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel, cts, state));
// Startup beep — queued onto downstreamChannel before the receive loop produces any audio.
short[] beep = MakeBeep(C.BeepHz, C.BeepSeconds, C.SampleRate);
Console.WriteLine("[beep]");
await downstreamChannel.Writer.WriteAsync(beep, cts.Token);
// Wait for beep to drain (200 ms beep + 100 ms margin) so it doesn't get captured
// by the mic and shipped to OpenAI.
Thread.Sleep((int)(C.BeepSeconds * 1000) + C.BeepDrainMs);
state.MicArmed = true;
Console.WriteLine("Speak now.");
try
{
doneFlag.Wait(cts.Token);
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
Console.WriteLine("[ws] closing");
}
finally
{
// Cancel any background work first so receive/send loops exit cleanly.
cts.Cancel();
if (ws.State == WebSocketState.Open || ws.State == WebSocketState.CloseReceived)
{
try
{
using var closeTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(2));
await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", closeTimeout.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[ws] close failed: {ex.Message}");
}
}
try { inputStream?.Stop(); } catch { }
try { outputStream?.Stop(); } catch { }
try { inputStream?.Dispose(); } catch { }
try { outputStream?.Dispose(); } catch { }
try { ws.Dispose(); } catch { }
try { PortAudio.Terminate(); } catch { }
Console.WriteLine("bye");
}
static string LoadApiKey()
{
string home = Environment.GetEnvironmentVariable("HOME") ?? "/root";
string path = Path.Combine(home, ".openai_key");
if (!File.Exists(path))
{
Console.Error.WriteLine($"[error] API key file not found at {path}");
Environment.Exit(2);
}
// Non-fatal mode-600 check.
try
{
var mode = File.GetUnixFileMode(path);
var leaky = mode & (UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute
| UnixFileMode.OtherRead | UnixFileMode.OtherWrite | UnixFileMode.OtherExecute);
if (leaky != UnixFileMode.None)
Console.Error.WriteLine($"[warn] {path} mode includes group/other perms ({mode}); expected 600");
}
catch (PlatformNotSupportedException) { /* not Unix */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[warn] could not check {path} mode: {ex.Message}");
}
string key = File.ReadAllText(path).Trim();
if (string.IsNullOrEmpty(key))
{
Console.Error.WriteLine($"[error] API key file empty");
Environment.Exit(2);
}
if (!key.StartsWith("sk-"))
{
Console.Error.WriteLine($"[error] API key doesn't start with sk-");
Environment.Exit(2);
}
return key;
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static async Task SendSessionUpdate(ClientWebSocket ws, CancellationToken ct)
{
var msg = new
{
type = "session.update",
session = new
{
modalities = new[] { "audio", "text" },
instructions = C.Instructions,
voice = C.Voice,
input_audio_format = "pcm16",
output_audio_format = "pcm16",
turn_detection = new
{
type = "server_vad",
threshold = 0.5,
prefix_padding_ms = 300,
silence_duration_ms = 500,
create_response = true,
},
},
};
byte[] bytes = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(bytes), WebSocketMessageType.Text, endOfMessage: true, ct);
}
static Stream OpenInputStream(int device, Channel<short[]> upstream, SharedState state)
{
var inParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (!state.MicArmed || state.StopSending)
return StreamCallbackResult.Continue;
if (frameCount != C.MicBlockFrames)
{
Console.Error.WriteLine($"[status] unexpected input frameCount={frameCount} (want {C.MicBlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[C.MicBlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, C.MicBlockFrames * sizeof(short), C.MicBlockFrames * sizeof(short));
}
if (!upstream.Writer.TryWrite(frame))
Console.Error.WriteLine("[status] upstream behind, dropping mic frame");
return StreamCallbackResult.Continue;
};
return new Stream(inParams, null, C.SampleRate, C.MicBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static Stream OpenOutputStream(int device, Channel<short[]> downstream, SharedState state, ManualResetEventSlim doneFlag)
{
var outParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int fill = 0;
while (fill < frameCount)
{
// Pull next chunk if we have no working one.
if (state.CurrentChunk == null)
{
if (!downstream.Reader.TryRead(out var nextChunk))
break; // ring empty
state.CurrentChunk = nextChunk;
state.ChunkOffset = 0;
}
// Local non-null alias so the compiler doesn't warn on dereference inside `fixed`.
short[] chunk = state.CurrentChunk!;
int slack = (int)frameCount - fill;
int remaining = chunk.Length - state.ChunkOffset;
int take = Math.Min(remaining, slack);
unsafe
{
short* dst = (short*)output.ToPointer();
fixed (short* src = &chunk[state.ChunkOffset])
Buffer.MemoryCopy(src, dst + fill, take * sizeof(short), take * sizeof(short));
}
state.ChunkOffset += take;
fill += take;
if (state.ChunkOffset >= chunk.Length)
state.CurrentChunk = null;
}
unsafe
{
short* dst = (short*)output.ToPointer();
for (int i = fill; i < (int)frameCount; i++) dst[i] = 0;
}
// Completion = working chunk drained AND channel empty AND writer completed.
if (state.CurrentChunk == null
&& downstream.Reader.Count == 0
&& downstream.Reader.Completion.IsCompleted)
{
doneFlag.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
return new Stream(null, outParams, C.SampleRate, C.OutBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static async Task UpstreamSendLoop(ClientWebSocket ws, Channel<short[]> upstream, CancellationTokenSource cts, SharedState state)
{
try
{
await foreach (var frame in upstream.Reader.ReadAllAsync(cts.Token))
{
if (state.StopSending) continue;
ReadOnlySpan<byte> bytes = MemoryMarshal.AsBytes(frame.AsSpan());
string base64 = Convert.ToBase64String(bytes);
var msg = new { type = "input_audio_buffer.append", audio = base64 };
byte[] json = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(json), WebSocketMessageType.Text, endOfMessage: true, cts.Token);
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] upstream send: {ex.Message}");
cts.Cancel();
}
}
static async Task ReceiveLoop(ClientWebSocket ws, Channel<short[]> downstream, CancellationTokenSource cts, SharedState state)
{
var transcript = new StringBuilder();
var buffer = new byte[4096];
bool firstDeltaSeen = false;
DateTime speechStoppedTime = DateTime.MinValue;
using var msgBuffer = new MemoryStream();
try
{
while (!cts.IsCancellationRequested)
{
msgBuffer.SetLength(0);
WebSocketReceiveResult result;
do
{
result = await ws.ReceiveAsync(new ArraySegment<byte>(buffer), cts.Token);
if (result.MessageType == WebSocketMessageType.Close) return;
msgBuffer.Write(buffer, 0, result.Count);
} while (!result.EndOfMessage);
msgBuffer.Position = 0;
using var doc = JsonDocument.Parse(msgBuffer);
string type = doc.RootElement.GetProperty("type").GetString() ?? "";
switch (type)
{
case "session.created":
{
string? sid = doc.RootElement.TryGetProperty("session", out var sess)
&& sess.TryGetProperty("id", out var idEl) ? idEl.GetString() : null;
Console.WriteLine($"[ws] session.created {sid}");
break;
}
case "session.updated":
Console.WriteLine("[ws] session.updated — session pinned");
break;
case "input_audio_buffer.speech_started":
Console.WriteLine("[vad] speech started");
break;
case "input_audio_buffer.speech_stopped":
Console.WriteLine("[vad] speech stopped");
speechStoppedTime = DateTime.UtcNow;
state.StopSending = true;
break;
case "input_audio_buffer.committed":
Console.WriteLine("[ws] input_audio_buffer.committed");
break;
case "response.created":
Console.WriteLine("[ws] response.created");
break;
case "response.audio_transcript.delta":
if (doc.RootElement.TryGetProperty("delta", out var tEl))
transcript.Append(tEl.GetString());
break;
case "response.audio.delta":
{
string b64 = doc.RootElement.GetProperty("delta").GetString() ?? "";
byte[] audioBytes = Convert.FromBase64String(b64);
short[] samples = new short[audioBytes.Length / 2];
Buffer.BlockCopy(audioBytes, 0, samples, 0, audioBytes.Length);
if (!firstDeltaSeen)
{
firstDeltaSeen = true;
double elapsed = speechStoppedTime == DateTime.MinValue
? -1
: (DateTime.UtcNow - speechStoppedTime).TotalSeconds;
Console.WriteLine($"[audio] first delta received (t = {elapsed:0.00} s since speech_stopped)");
}
await downstream.Writer.WriteAsync(samples, cts.Token);
break;
}
case "response.audio.done":
Console.WriteLine("[ws] response.audio.done");
break;
case "response.done":
state.NoMoreDeltas = true;
downstream.Writer.Complete();
Console.WriteLine("[ws] response.done");
Console.WriteLine($"[transcript] {transcript}");
break;
case "error":
Console.Error.WriteLine($"[error] WS error event: {doc.RootElement.GetRawText()}");
cts.Cancel();
break;
default:
Console.Error.WriteLine($"[ws] ignored type={type}");
break;
}
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] receive loop: {ex.Message}");
cts.Cancel();
}
}
sealed class SharedState
{
public volatile bool MicArmed;
public volatile bool StopSending;
public volatile bool NoMoreDeltas;
// Output-callback-thread-local; do NOT touch from anywhere else.
public short[]? CurrentChunk;
public int ChunkOffset;
}
static class C
{
public const int SampleRate = 24_000;
public const int Channels = 1;
public const uint MicBlockFrames = 1920; // 40 ms @ 24 kHz
public const uint OutBlockFrames = 1024; // ~43 ms @ 24 kHz
public const string ModelName = "gpt-realtime";
public const string Voice = "alloy";
public const string Instructions = "Reply in one short sentence.";
public const double BeepHz = 880.0;
public const double BeepSeconds = 0.2;
public const int BeepDrainMs = 100;
public const int UpstreamChannelCapacity = 64;
public const int DownstreamChannelCapacity = 64;
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}
```
- [ ] **Step 2: Verify the project builds cleanly**
```bash
dotnet build tests/04a-realtime-oneshot-cs -c Release
```
Expected: `Build succeeded` with 0 errors. A handful of nullable-reference warnings on `state.CurrentChunk` access inside the `unsafe` block is fine if it appears; the runtime checks above ensure non-null.
If `<AllowUnsafeBlocks>` is missing from the csproj, the build will fail with `CS0227: Unsafe code may only appear if compiling with /unsafe`. Confirm Task 1's csproj has `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`.
If you see `CS0246: The type or namespace 'Stream' could not be found` or a clash with `System.IO.Stream`, confirm the `using Stream = PortAudioSharp.Stream;` line is at the top of `Program.cs`.
If you see `error CS8803: Top-level statements must precede namespace and type declarations` or `CS8805: Program using top-level statements must be an entry point`, the file ordering is wrong. The exact required order in `Program.cs` is:
1. `using` directives
2. Top-level statements (the imperative body — `Environment.SetEnvironmentVariable`, `apiKey = LoadApiKey()`, `PortAudio.Initialize`, the `try {} finally {}` block)
3. Top-level static method declarations (`LoadApiKey`, `FindUsbDevice`, `MakeBeep`, `SendSessionUpdate`, `OpenInputStream`, `OpenOutputStream`, `UpstreamSendLoop`, `ReceiveLoop`)
4. Top-level type declarations (`sealed class SharedState`, `static class C`, `static class Libc`) — NOT wrapped in a `namespace` block.
The file above follows this order; do not reorder.
- [ ] **Step 3: Verify `dotnet publish` produces a runnable binary**
```bash
dotnet publish tests/04a-realtime-oneshot-cs -c Release -r linux-arm64 --self-contained -o /tmp/probe-cs-4a-out
ls /tmp/probe-cs-4a-out/Probe
```
Expected:
- `Probe` (executable) exists.
- No `models/` directory needed.
- [ ] **Step 4: Commit**
```bash
git add tests/04a-realtime-oneshot-cs/Program.cs
git commit -m "Test 4a: WS plumbing + audio streams + server VAD round-trip"
```
---
### Task 3: Write `bin/probe-cs-4a` deploy script with API key precondition
**Files:**
- Create: `bin/probe-cs-4a`
This is `bin/probe-cs-3` with three string swaps (`03-full-cycle-cs``04a-realtime-oneshot-cs`, `probe-cs-3``probe-cs-4a`, `/tmp/probe-cs-3-out``/tmp/probe-cs-4a-out`) plus ONE new precondition step: check that `~/.openai_key` exists on the Pi before publishing. This catches the silent "key missing" failure mode at the deploy step instead of 30 s into publish + scp.
- [ ] **Step 1: Write the deploy script**
Create `bin/probe-cs-4a` with exactly this content:
```bash
#!/usr/bin/env bash
set -euo pipefail
PI_HOST="${PI_HOST:-192.168.50.115}"
PI_USER="${PI_USER:-pi}"
PI_PASS="${PI_PASS:-assistant}"
PUBLISH_DIR="${PUBLISH_DIR:-/tmp/probe-cs-4a-out}"
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"
echo ">> precondition: ~/.openai_key present on Pi"
if ! sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'test -f ~/.openai_key && test -r ~/.openai_key'; then
echo "ERROR: ~/.openai_key missing or unreadable on Pi."
echo "Create it on the Pi with:"
echo " sshpass -p '$PI_PASS' ssh $PI_USER@$PI_HOST \\"
echo " 'umask 077 && printf \"%s\\n\" \"sk-PASTE-YOUR-KEY\" > ~/.openai_key'"
exit 1
fi
echo " OK."
echo ">> dotnet publish (linux-arm64, self-contained)"
dotnet publish tests/04a-realtime-oneshot-cs \
-c Release -r linux-arm64 --self-contained \
-o "$PUBLISH_DIR"
echo ">> scp to $PI_USER@$PI_HOST:~/probe-cs-4a/ (wipe-and-replace)"
sshpass -p "$PI_PASS" ssh -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a'
sshpass -p "$PI_PASS" scp -r "$PUBLISH_DIR/." \
"$PI_USER@$PI_HOST:~/probe-cs-4a/"
echo ">> ssh + run on Pi (Ctrl-C from this terminal stops the probe)"
sshpass -p "$PI_PASS" ssh -t -o StrictHostKeyChecking=accept-new \
"$PI_USER@$PI_HOST" 'cd ~/probe-cs-4a && chmod +x Probe && ./Probe'
```
- [ ] **Step 2: Make it executable**
```bash
chmod +x bin/probe-cs-4a
```
- [ ] **Step 3: Commit**
```bash
git add bin/probe-cs-4a
git commit -m "Test 4a: bin/probe-cs-4a deploy script with ~/.openai_key precondition"
```
---
### Task 4: One-time Pi setup — place `~/.openai_key` on the Pi
**Files:** (none — this is a one-time remote setup task)
This task only runs once per Pi. If the file already exists from a previous deploy or another test, skip steps 12 and verify with step 3.
- [ ] **Step 1: Ask the user for an API key**
Before running this task, the user must have an OpenAI API key with Realtime API access (`sk-…`). If the user hasn't given it to you yet, ask:
> "I need your OpenAI API key (`sk-…`) to write it to `~/.openai_key` on the Pi. Paste it and I'll redact it from the shell history immediately. Do not paste it into any other context."
The key the user provides MUST be treated as secret and MUST NOT appear in commit messages, file contents anywhere in the repo, console echoes, or logs. The only place it goes is the file on the Pi.
- [ ] **Step 2: Write the key to the Pi**
Replace `sk-PASTE-YOUR-KEY-HERE` below with the actual key value. The `umask 077` ensures the file is created with mode 600 (rw for owner only). The `printf "%s\n"` appends exactly one newline.
```bash
sshpass -p 'assistant' ssh -o StrictHostKeyChecking=accept-new \
pi@192.168.50.115 \
'umask 077 && printf "%s\n" "sk-PASTE-YOUR-KEY-HERE" > ~/.openai_key'
```
- [ ] **Step 3: Verify file exists and is mode 600**
```bash
sshpass -p 'assistant' ssh pi@192.168.50.115 \
'ls -l ~/.openai_key && wc -c ~/.openai_key'
```
Expected output (mode `-rw-------`, owner `pi`, size > 50 bytes):
```
-rw------- 1 pi pi <size> <date> /home/pi/.openai_key
<size> /home/pi/.openai_key
```
If mode is not `-rw-------`, fix with `chmod 600 ~/.openai_key`.
If size is suspiciously small (< 20 bytes), the key wasn't pasted correctly — re-do step 2.
(No commit — this step modifies state on the Pi, not the repo.)
---
### Task 5: Hardware verification on the Pi
This task is the actual pass/fail gate. No automated tests; verification is the human ear + console output. If anything in this task fails, do NOT mark the implementation complete — investigate and either fix in code or update the spec to reflect what actually works.
**Files:** (none — this is a runtime verification task)
- [ ] **Step 1: Confirm the Pi is reachable**
Run from workstation:
```bash
ping -c 2 192.168.50.115
```
Expected: 2/2 replies. If unreachable, stop and ask the user.
- [ ] **Step 2: Deploy + run the probe**
Run from workstation:
```bash
./bin/probe-cs-4a
```
Expected console output (abridged, with elision):
```
>> precondition: ~/.openai_key present on Pi
OK.
>> dotnet publish (linux-arm64, self-contained)
... (build output) ...
>> scp to pi@192.168.50.115:~/probe-cs-4a/ (wipe-and-replace)
>> ssh + run on Pi (Ctrl-C from this terminal stops the probe)
[ws] connecting to wss://api.openai.com/v1/realtime?model=gpt-realtime
[ws] connected
[ws] session.created sess_xxxxxxxxxxxx
[ws] session.updated — session pinned
Using device <N> ('USB Speaker Phone (LISTENAI ...)')
[beep]
Speak now.
```
At this point the probe is waiting for you to speak. Say a single short question into the USB Speaker Phone — for example: **"What's the capital of France?"** — at normal volume from ~1 m.
Continue with verification in step 3.
If the publish step errors with a runtime crash invoking `Libc.setenv`, confirm Task 2's `[DllImport]` line for `setenv` is present.
If the Pi run errors with `Cannot open shared library libportaudio.so`, the self-contained publish didn't bundle the native libs — confirm `<SelfContained>true</SelfContained>` and `<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>` in Task 1's csproj.
If `[error] WS connect failed: ... HTTP 401` appears, the API key was rejected. Re-check Task 4 step 3 and confirm the key on the Pi starts with `sk-` and the user pasted the right value.
- [ ] **Step 3: Verify the full conversational round-trip**
After saying your question in step 2, observe:
1. `[vad] speech started` appears in the console as you begin speaking.
2. `[vad] speech stopped` appears shortly after you stop speaking (within ~0.51 s, governed by `silence_duration_ms = 500` in session.update).
3. `[ws] input_audio_buffer.committed` and `[ws] response.created` follow.
4. `[audio] first delta received (t = X.XX s since speech_stopped)` appears with a small latency value (~12 s).
5. The assistant's spoken reply comes out of the USB Speaker Phone speaker — clearly, intelligibly, recognisable as English speech.
6. `[ws] response.audio.done` and `[ws] response.done` follow.
7. `[transcript] <the assistant's reply, as text>` appears — should match what you heard.
8. `[ws] closing` and `bye` appear; the probe exits.
**Pass criterion 1:** End-to-end round-trip works once. Probe exits 0.
**Pass criterion 2:** Response is intelligible (no clipping, no dropouts longer than one ~43 ms callback).
**Pass criterion 8:** Probe exits within ~500 ms of the last audio sample being heard.
- [ ] **Step 4: Check latency from the log line**
Look at the `[audio] first delta received (t = X.XX s since speech_stopped)` line printed in step 3.
**Pass criterion 3:** `X.XX` is under ~2 s. Slower is acceptable if logged here; over ~5 s is a fail.
- [ ] **Step 5: Scan for warnings**
Re-read the full console output. Confirm there are **no** lines of these forms during normal operation (between `[beep]` and `[ws] response.done`):
- `[status] upstream behind, dropping mic frame`
- `[status] input overflow`
- `[status] unexpected input frameCount=...`
- `[error] WS ...`
- `[ws] ignored type=...` (one or two of these are OK — the API may send events we don't handle; sustained occurrences during normal flow indicate a missing handler we should add)
**Pass criterion 4:** No `[status] upstream behind` warnings.
**Pass criterion 5:** No `[error] WS …` events.
**Pass criterion 6:** No PortAudio `InputOverflow` status flags.
- [ ] **Step 6: Test Ctrl-C clean exit (separate run)**
Restart the probe:
```bash
./bin/probe-cs-4a
```
When `Speak now.` appears, press Ctrl-C in the workstation terminal **without** speaking.
Expected:
```
^C
[ws] closing
bye
$
```
Verify with `echo $?` — exit code 0.
**Pass criterion 7:** Ctrl-C produces a clean exit (no stack trace, terminal returns to prompt, exit code 0 when invoked during the steady listening phase).
- [ ] **Step 7: Update `findings.md` with the outcome**
Add a new section `## Test 4a outcome (2026-06-12 — OpenAI Realtime one-shot voice round-trip in C#)` at the bottom of `findings.md` summarising:
- Pass / fail per criterion (18).
- Latency measured (the `X.XX` value from step 4).
- Any new gotchas discovered during implementation or hardware verification — especially:
- Any OpenAI Realtime API surface differences from the spec (event types not in our switch, session.update fields rejected, model name issues).
- Any audio-format quirks (24 kHz playback dropouts, sample-rate-mismatch artifacts).
- Any concurrency surprises (channel deadlocks, send-loop wedges).
- Any errors in our LoadApiKey / mode-600 check.
- Implications for 4b (what we still need to debug there vs. what 4a proved out).
- Reference paths (spec + plan + code + deploy script).
Use the existing Test 1, 2, and 3 outcome sections in `findings.md` as the template.
- [ ] **Step 8: Commit the findings update**
```bash
git add findings.md
git commit -m "Findings: Test 4a OpenAI Realtime one-shot voice round-trip outcome"
```
- [ ] **Step 9: Final status report**
Report to the user:
- Pass/fail per criterion (1 through 8 from spec).
- Any criteria that failed and what was discovered.
- The measured round-trip latency value.
- A one-line summary of whether Test 4a unblocks Test 4b (i.e. whether the OpenAI Realtime surface + audio plumbing is proven, so 4b can focus on wakeword + loop + 16↔24 kHz coexistence).
---
## Self-review notes
- **Spec coverage:**
- Goal (single-binary one-shot probe) → Tasks 1+2+3+5.
- `~/.openai_key` precondition + file-on-Pi storage → Task 3 (script) + Task 4 (one-time setup) + Task 2 (`LoadApiKey` with mode-600 warning).
- WS connect with `Authorization` + `OpenAI-Beta` headers → Task 2 (`ws.Options.SetRequestHeader` + `ConnectAsync`).
- `session.update` payload with `server_vad` → Task 2 (`SendSessionUpdate` helper, exact JSON shape).
- Mic input stream at 24 kHz with `_micArmed` gate → Task 2 (`OpenInputStream`, `state.MicArmed` check in callback).
- UpstreamSendLoop (base64 + `input_audio_buffer.append`) → Task 2 (`UpstreamSendLoop`).
- ReceiveLoop with `JsonDocument` switch on `type` → Task 2 (`ReceiveLoop`, complete handler table).
- Output stream drain-or-zero-fill + `_noMoreDeltas` + `Reader.Completion.IsCompleted` completion → Task 2 (`OpenOutputStream` callback).
- Startup beep through `downstreamChannel` → Task 2 (lines after `outputStream.Start()`).
- `ManualResetEventSlim doneFlag` + clean shutdown via `finally` → Task 2 (try/finally block).
- Ctrl-C → `Console.CancelKeyPress``cts.Cancel``OperationCanceledException` → finally → exit 0 → Tasks 2 + 5 step 6.
- `bin/probe-cs-4a` with ssh -t + wipe-and-replace + key precondition → Task 3.
- Pass criteria 18 → Task 5 steps 3, 4, 5, 6.
- Findings update → Task 5 steps 78.
- **Placeholder scan:** no TBD/TODO; every code step shows the actual code; every command shows expected output. The single user-substitution placeholder is `sk-PASTE-YOUR-KEY-HERE` in Task 4 step 2, which is explicitly called out as a replacement target.
- **Type consistency:**
- `SharedState` fields (`MicArmed`, `StopSending`, `NoMoreDeltas`, `CurrentChunk`, `ChunkOffset`) referenced consistently from `OpenInputStream`, `OpenOutputStream`, `UpstreamSendLoop`, `ReceiveLoop`, and the top-level statements.
- `Channel<short[]>` used for both upstream and downstream; the `short[]` element type is consistent everywhere.
- `ManualResetEventSlim doneFlag` passed by reference into `OpenOutputStream` so the callback can `Set()` it; main thread awaits it via `Wait(cts.Token)`.
- `CancellationTokenSource cts` is the single source of cancellation across all three loops + the main thread.
- **Scope:** the spec is a single subsystem (one probe). One plan is the right granularity.
@@ -0,0 +1,474 @@
# Test 4a — OpenAI Realtime API one-shot voice round-trip (C#)
A throwaway C# probe that has a single round-trip spoken conversation with the OpenAI Realtime API:
> Boot → beep → user speaks one sentence → server VAD ends the turn → assistant audio reply plays back through the same USB Speaker Phone → exit 0.
No wakeword. No loop. No state machine. One process, runs once, dies.
Test 4a's job is to debug the OpenAI Realtime API surface (auth, WS handshake, `session.update` payload, event JSON shapes, base64 audio decode, server VAD endpointing, streaming-length output-stream cleanup) **in isolation** from the harder concurrency story of Test 4b (wakeword + loop + 16↔24 kHz coexistence).
Read `findings.md` (§ "C# probe outcome", § "C# wakeword probe outcome", § "C# full-cycle outcome") before reading this. Everything below assumes those outcomes as ground truth. The patterns lifted directly from Test 3 are: `Libc.setenv`, the `using Stream = PortAudioSharp.Stream;` import disambiguation, `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`, the fire-and-forget output stream lifetime (Start under try/catch BEFORE the cleanup task launches), `FindUsbDevice`, the `MakeBeep` helper, `bin/probe-cs-3`'s wipe-and-replace publish shape with `ssh -t`.
## Goal
A single-binary C# probe that:
1. Reads the OpenAI API key from `~/.openai_key` on the Pi (mode 600).
2. Connects to `wss://api.openai.com/v1/realtime?model=gpt-realtime` over a WebSocket with `Authorization: Bearer …` + `OpenAI-Beta: realtime=v1` headers.
3. Sends one `session.update` event pinning audio formats, voice, instructions, and `turn_detection: server_vad`.
4. Plays a startup beep through the USB Speaker Phone.
5. Opens a 24 kHz mono Int16 input stream on the same device; streams mic audio to OpenAI as `input_audio_buffer.append` events.
6. Waits for the server VAD to fire `input_audio_buffer.speech_stopped` (server then auto-commits the buffer and creates the response).
7. Stops sending mic audio at that point. Receives `response.audio.delta` chunks, base64-decodes them, plays them through a 24 kHz output stream on the same device.
8. On `response.done` AND the output buffer drained, closes the WebSocket cleanly, terminates PortAudio, exits 0.
9. Exits cleanly on Ctrl-C from the workstation (SIGINT propagated via `ssh -t`) at any stage.
## Non-goals
- **No wakeword.** No `WakewordModel.cs`. No 16 kHz audio path anywhere in 4a. (Test 4b layers that on top of this proven base.)
- **No state machine, no loop.** Probe runs once and exits.
- **No barge-in.** Once `speech_stopped` fires, mic frames are dropped on the floor. The user cannot interrupt the assistant in 4a.
- **No client-side resampling.** Mic = 24 kHz, output = 24 kHz, OpenAI = 24 kHz pcm16. PortAudio plug-resamples 48↔24 at ALSA on the USB device.
- **No WAV capture.** Logging is stdout/stderr only.
- **No retry on WS errors, no reconnect, no health check.** Fail loud, exit non-zero, log enough to diagnose.
- **No automated tests.** Hardware verification only.
## Architecture
```
┌──────────────────────────────────────┐
│ ClientWebSocket │
│ wss://api.openai.com/v1/realtime? │
│ model=gpt-realtime │
│ Headers: Authorization: Bearer … │
│ OpenAI-Beta: realtime=v1 │
└────┬──────────────────────────┬──────┘
│ SendAsync │ ReceiveAsync
▼ ▼
┌──────────────────────────┐ ┌──────────────────────────────┐
│ UpstreamSendLoop │ │ ReceiveLoop (Task.Run) │
│ (Task.Run) │ │ while (!cts.Cancel) { │
│ drain upstreamChannel: │ │ msg = ReceiveAsync(…) │
│ pull short[1920] │ │ doc = JsonDocument.Parse │
│ → base64 │ │ switch (doc["type"]) │
│ → input_audio_ │ │ session.created → log │
│ buffer.append │ │ session.updated → log │
│ → ws.SendAsync │ │ speech_started → log │
│ exits when │ │ speech_stopped → │
│ _stopSending && empty│ │ _stopSending = true │
└──────────┬───────────────┘ │ response.audio.delta: │
│ (FullMode.Wait) │ base64 → short[] │
│ │ → downstreamChannel │
▲ │ .Writer.WriteAsync │
│ short[1920] │ audio_transcript.delta:│
│ (40 ms @ 24 kHz) │ append to log buf │
┌──────────┴───────────────┐ │ response.done → │
│ PortAudio input stream │ │ _noMoreDeltas = true │
│ 24 kHz mono Int16, │ │ downstreamChannel │
│ 1920 frames/cb │ │ .Writer.Complete │
│ callback: │ │ error → log + Cancel │
│ if !_micArmed: discard│ └────────────┬─────────────────┘
│ elif _stopSending: │ │
│ discard │ ▼
│ else: │ ┌──────────────────────────────┐
│ copy → short[] │ │ Channel<short[]> │
│ TryWrite to │ │ (bounded cap 64, │
│ upstreamChannel │ │ FullMode.Wait) │
└──────────────────────────┘ └────────────┬─────────────────┘
┌──────────────────────────────────┐
│ PortAudio output stream │
│ 24 kHz mono Int16, │
│ 1024 frames/cb │
│ callback: │
│ fill = 0 │
│ while fill < frameCount: │
│ if currentChunk == null: │
│ try pull short[] or break │
│ take = min(remaining, slack)│
│ memcpy chunk → output │
│ zero-fill remaining │
│ if currentChunk == null │
│ && Reader.Count == 0 │
│ && Completion.IsCompleted: │
│ doneFlag.Set() │
│ return Complete │
│ return Continue │
└────────────┬─────────────────────┘
main thread: doneFlag.Wait
→ ws.CloseAsync(NormalClosure)
→ input/output stream.Stop
→ PortAudio.Terminate
→ exit 0
```
One process. Three managed threads (main + UpstreamSendLoop + ReceiveLoop) plus PortAudio's input and output audio threads (managed by the library).
**Shared mutable state — the full list, small on purpose:**
- `Channel<short[]> upstreamChannel`, bounded 64, `FullMode.Wait`, single-writer (mic callback), single-reader (UpstreamSendLoop). Holds mic frames waiting to be base64'd and sent.
- `Channel<short[]> downstreamChannel`, bounded 64, `FullMode.Wait`, **multi-writer** (main thread writes the startup beep; ReceiveLoop writes response audio chunks; temporally separated but Channel<T>'s `SingleWriter` flag is a contract not a runtime check, so keep it `false`), single-reader (output callback). Holds decoded audio chunks waiting to be played. Its `Writer.Complete()` (called by ReceiveLoop on `response.done`) is the WS-stream-end signal for the output callback.
- `volatile bool _micArmed`: starts `false`; set `true` after the startup beep has been queued and drained. Mic frames before this point are discarded so the beep doesn't get captured and shipped to OpenAI.
- `volatile bool _stopSending`: set by ReceiveLoop on `input_audio_buffer.speech_stopped`. Read by both the mic callback (which then discards) and UpstreamSendLoop (which then drains its queue and exits).
- `volatile bool _noMoreDeltas`: set by ReceiveLoop on `response.done`. Not strictly required because `downstreamChannel.Writer.Complete()` carries the same signal — but kept as a redundant fast path that the output callback can check without touching the channel.
- `ManualResetEventSlim doneFlag`: set by the output callback when it has drained the channel after completion. Awaited by main.
- `CancellationTokenSource cts`: wired to `Console.CancelKeyPress` + any WS error. Cancels both Task.Run loops; main's `doneFlag.Wait(cts.Token)` honours it.
No locks. No `Interlocked`. The bounded `Channel<T>` handles all the cross-thread audio handoff with the right backpressure semantics.
## WebSocket lifecycle
Six discrete phases, all in `Program.cs` top-level statements + the two `Task.Run` loops:
1. **Load API key.** `LoadApiKey()` helper reads `~/.openai_key`, trims whitespace. Mode 600 verified — non-fatal warning if looser. Fatal error (exit 2) if the file is missing, empty, or doesn't start with `sk-`.
2. **Construct `ClientWebSocket`** with two request headers: `Authorization: Bearer <key>` and `OpenAI-Beta: realtime=v1`. `ConnectAsync` to `wss://api.openai.com/v1/realtime?model=gpt-realtime` with `cts.Token`. On failure (DNS, TLS, 401, 403, 429, 5xx), log the exception message + any HTTP status carried on `WebSocketException.WebSocketErrorCode` / inner `HttpRequestException.StatusCode`; exit 3.
3. **Launch ReceiveLoop** (`Task.Run`) before sending anything, so we don't miss `session.created`.
4. **Send `session.update`.** First client message; pins the session config (see "Client → server events" below). Don't strictly block on `session.updated` — the receive loop logs it asynchronously — but the spec expects it within ~200 ms.
5. **Open + start PortAudio streams.** Input stream first (24 kHz mono Int16, 1920 frames/cb), then output stream (24 kHz mono Int16, 1024 frames/cb). Both opened on `FindUsbDevice()`. Launch UpstreamSendLoop (`Task.Run`).
6. **Startup beep + arm mic.** Queue a 200 ms 880 Hz beep (`MakeBeep` lifted from Test 3, regenerated at 24 kHz) onto `downstreamChannel`. Sleep ~300 ms (beep + drain margin). Set `_micArmed = true`. Print `Speak now.` to stdout.
7. **Wait for `doneFlag`** on the main thread (with `cts.Token` for Ctrl-C). On wake-up: `ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", …)` with a 2 s timeout token. Stop input stream, stop output stream, terminate PortAudio, exit 0.
## Client → server event payloads
Three send-once messages on the main thread, plus a stream of `input_audio_buffer.append` from UpstreamSendLoop. All written with `JsonSerializer.Serialize` over `Dictionary<string, object?>` (lightest thing that doesn't require typed records for a probe).
```json
// (1) Pinned session config — first thing sent after WS connect.
{
"type": "session.update",
"session": {
"modalities": ["audio", "text"],
"instructions": "Reply in one short sentence.",
"voice": "alloy",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"prefix_padding_ms": 300,
"silence_duration_ms": 500,
"create_response": true
}
}
}
// (2) Per mic frame (~40 ms cadence), from UpstreamSendLoop:
{ "type": "input_audio_buffer.append", "audio": "<base64-pcm16-24k>" }
```
No `conversation.item.create` and no `response.create` from the client — server VAD's `create_response: true` makes the server commit and create the response automatically on `speech_stopped`.
`instructions: "Reply in one short sentence."` keeps the response audio bounded (~2-4 s of speech) so a probe run is fast to verify by ear and cheap on quota.
`voice: "alloy"` is the long-standing default voice and is documented as universally available for raw WS sessions.
## Server → client event handlers
ReceiveLoop uses a 4 KB receive buffer + standard `ReceiveAsync` loop with `EndOfMessage` accumulation into a `MemoryStream`. On message end, `JsonDocument.Parse`. Switch on the top-level `type` property's string value:
| `type` | Action |
|-|-|
| `session.created` | log `[ws] session.created <session_id>` |
| `session.updated` | log `[ws] session.updated — session pinned` |
| `input_audio_buffer.speech_started` | log `[vad] speech started` |
| `input_audio_buffer.speech_stopped` | log `[vad] speech stopped`; set `_stopSending = true` |
| `input_audio_buffer.committed` | log only — server confirms it sealed the buffer |
| `response.created` | log only |
| `response.audio_transcript.delta` | append `delta` to an accumulating `StringBuilder` |
| `response.audio.delta` | base64-decode `delta` → bytes → reinterpret as `short[]``await downstreamChannel.Writer.WriteAsync(samples, cts.Token)`. The `await` is the backpressure path: if the channel is full, this blocks the receive loop, which blocks the WS receive buffer — correct behavior for a slow speaker. |
| `response.audio.done` | log only |
| `response.done` | set `_noMoreDeltas = true`; `downstreamChannel.Writer.Complete()`; print the accumulated transcript |
| `error` | log the full event JSON; `cts.Cancel()` |
| anything else | log `[ws] ignored type=<…>` once (or at debug verbosity), continue |
## Audio buffering — input
- Mic callback runs every 40 ms (1920 samples @ 24 kHz mono Int16, matches `framesPerBuffer = 1920`).
- Callback allocates `short[1920]`, `Buffer.MemoryCopy`s the input pointer, `upstreamChannel.Writer.TryWrite(frame)`.
- On TryWrite failure (channel full = UpstreamSendLoop is behind): log `[status] upstream behind, dropping mic frame`, continue.
- Discard frames when `_micArmed == false` (during startup beep) or `_stopSending == true` (after `speech_stopped`).
- On the `InputOverflow` status flag from PortAudio: log `[status] input overflow`, continue.
UpstreamSendLoop pseudocode:
```
await ReceiveLoop signals session.updated (or just optimistically loop from start)
while (!cts.IsCancellationRequested):
if _stopSending && upstreamChannel.Reader.Count == 0: break
frame = await upstreamChannel.Reader.ReadAsync(cts.Token)
if _stopSending: continue // drop frames captured after server VAD cutoff
bytes = MemoryMarshal.AsBytes(frame.AsSpan())
base64 = Convert.ToBase64String(bytes)
json = JsonSerializer.SerializeToUtf8Bytes(new {
type = "input_audio_buffer.append",
audio = base64
})
await ws.SendAsync(json, WebSocketMessageType.Text, endOfMessage: true, cts.Token)
```
## Audio buffering — output
`downstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(64) { FullMode = BoundedChannelFullMode.Wait, SingleWriter = false, SingleReader = true })`. `SingleWriter = false` because the main thread queues the startup beep AND ReceiveLoop queues response audio chunks (temporally separated, but the flag is a hard contract).
Capacity rationale: 64 chunks × ~40-80 ms per `response.audio.delta` ≈ 2.5-5 s of buffered audio. Comfortable headroom against network jitter without holding silly amounts of memory.
ReceiveLoop writes `short[]` chunks of varying length (one per `response.audio.delta` event). On `response.done`, calls `downstreamChannel.Writer.Complete()`.
Output callback's job is to fill `frameCount` shorts every callback (1024 frames = ~43 ms @ 24 kHz). It maintains a `short[]? _currentChunk` + `int _chunkOffset` across calls. Pseudocode:
```
int fill = 0;
while (fill < frameCount) {
if (_currentChunk == null) {
if (!downstreamChannel.Reader.TryRead(out _currentChunk)) break; // drained
_chunkOffset = 0;
}
int slack = frameCount - fill;
int remaining = _currentChunk.Length - _chunkOffset;
int take = Math.Min(remaining, slack);
unsafe {
fixed (short* src = &_currentChunk[_chunkOffset])
Buffer.MemoryCopy(src, output + fill, take * 2, take * 2);
}
_chunkOffset += take;
fill += take;
if (_chunkOffset >= _currentChunk.Length) _currentChunk = null;
}
// Zero-fill the unfilled tail (underrun mid-response, or post-completion drain).
for (int i = fill; i < frameCount; i++) output[i] = 0;
// Completion check: drained AND writer completed AND no working chunk left.
if (_currentChunk == null
&& downstreamChannel.Reader.Count == 0
&& downstreamChannel.Reader.Completion.IsCompleted)
{
doneFlag.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
```
**Underrun behavior: zero-fill, return Continue.** If `response.done` hasn't arrived yet and the next `response.audio.delta` is just in flight, we play silence for one callback (~43 ms) instead of starving — the stream stays armed, no click on resume.
**This is the streaming-length cleanup discipline** that replaces Test 3's "scale `done.Wait` timeout to the buffer length" trick. Test 3 knew the playback wall-clock up front (`recordBuf.Length / SampleRate`); Test 4a does not, because the assistant audio length is unknown a priori. So the signaling moves from "wait this many seconds" to "wait until (writer-completed AND channel-empty AND working-chunk-empty)." The output callback is the single oracle for this triple condition.
**Startup beep wrinkle:** the beep is queued onto `downstreamChannel` BEFORE the receive loop starts producing. The output callback drains it just like any other audio. The completion check is safe because `Writer.Complete()` is only ever called on `response.done` — the beep getting drained does NOT cause an early completion firing.
## Components and files
```
tests/04a-realtime-oneshot-cs/
├── Probe.csproj
│ <TargetFramework>net9.0</TargetFramework>
│ <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
│ <SelfContained>true</SelfContained>
│ <PublishSingleFile>false</PublishSingleFile>
│ <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
│ <InvariantGlobalization>true</InvariantGlobalization>
│ <Nullable>enable</Nullable>
│ <ImplicitUsings>enable</ImplicitUsings>
│ RootNamespace=RealtimeOneShotProbe, AssemblyName=Probe
│ PackageReference PortAudioSharp2 Version="1.0.6"
│ (no Microsoft.ML.OnnxRuntime — no wakeword in 4a)
│ System.Net.WebSockets + System.Text.Json come from the BCL; no package.
├── Program.cs ~300 lines, top-level statements:
│ 1. Libc.setenv("PA_ALSA_PLUGHW", "1", 1) + Environment mirror
│ (no ORT env vars — no onnxruntime in 4a)
│ 2. PortAudio.Initialize
│ 3. apiKey = LoadApiKey()
│ 4. var cts = new CancellationTokenSource();
│ Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
│ 5. var ws = new ClientWebSocket();
│ ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
│ ws.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");
│ await ws.ConnectAsync(uri, cts.Token);
│ 6. Channel<short[]> upstreamChannel, downstreamChannel (both bounded 64)
│ ManualResetEventSlim doneFlag = new(false);
│ 7. Task receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel,
│ cts, _stopSending, _noMoreDeltas, transcriptBuilder));
│ 8. Send session.update (JsonSerializer.SerializeToUtf8Bytes over a dict)
│ 9. int device = FindUsbDevice();
│ 10. Open + start input stream (24 kHz, 1920 frames/cb)
│ 11. Open + start output stream (24 kHz, 1024 frames/cb)
│ 12. Task sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel,
│ cts, _stopSending));
│ 13. Queue MakeBeep(880, 0.2, 24000) onto downstreamChannel; sleep 300 ms;
│ _micArmed = true; Console.WriteLine("Speak now.");
│ 14. try { doneFlag.Wait(cts.Token); }
│ catch (OperationCanceledException) { /* Ctrl-C path */ }
│ 15. finally:
│ ws.CloseAsync(NormalClosure, "done", linkedCtsWith2sTimeout.Token)
│ inputStream.Stop; outputStream.Stop;
│ inputStream.Dispose; outputStream.Dispose;
│ PortAudio.Terminate;
│ Helpers (all in Program.cs):
│ - LoadApiKey() — reads ~/.openai_key, perms-warns, sanity
│ - MicCallback — discards unless _micArmed && !_stopSending
│ - OutputCallback — drain-or-zero-fill loop above
│ - MakeBeep(hz, sec, rate) — sin wave Int16 (lifted from Test 3, retuned 24 kHz)
│ - UpstreamSendLoop(...)
│ - ReceiveLoop(...) — JsonDocument switch on "type"
│ - FindUsbDevice() — lifted from Test 3 verbatim
│ - static class Libc { setenv }
└── (no models/ dir, no WakewordModel.cs)
bin/probe-cs-4a Copy of bin/probe-cs-3 with substitutions:
tests/03-full-cycle-cs → tests/04a-realtime-oneshot-cs
~/probe-cs-3 → ~/probe-cs-4a
PUBLISH_DIR /tmp/probe-cs-3-out → /tmp/probe-cs-4a-out
ssh -t retained (Ctrl-C propagation).
Wipe-and-replace publish dir retained (partial-deploy
footgun from Test 3).
Adds ONE new precondition step before publishing:
sshpass -p "$PI_PASS" ssh "$PI_USER@$PI_HOST" \
'test -f ~/.openai_key && test -r ~/.openai_key' \
|| { echo "Create ~/.openai_key on the Pi (chmod 600),
paste your sk-... key into it, then re-run."; \
exit 1; }
This catches the silent "key missing" failure mode at
the deploy step rather than 30 s into the publish + scp.
```
## Concurrency model
Five concurrent threads of execution in steady state:
1. **Main thread.** Setup, session.update send, beep + arm, `doneFlag.Wait`, teardown via `finally`.
2. **ReceiveLoop** (`Task.Run`). Pulls WS messages, dispatches on `type`. Single writer of `downstreamChannel`, `_stopSending`, `_noMoreDeltas`, transcript builder. Calls `cts.Cancel()` on error.
3. **UpstreamSendLoop** (`Task.Run`). Pulls mic frames from `upstreamChannel`, JSON-serializes `input_audio_buffer.append`, calls `ws.SendAsync`. Single writer of the WS send stream.
4. **PortAudio input audio thread** (managed by the library). Runs the mic callback every 40 ms. Allocates `short[1920]`, memcpy's the input, `upstreamChannel.Writer.TryWrite(frame)`. Never blocks.
5. **PortAudio output audio thread** (managed by the library). Runs the output callback every 43 ms. Drains `downstreamChannel` into the output buffer; zero-fills underruns; sets `doneFlag` on completion. Never blocks.
Note that `ws.SendAsync` and `ws.ReceiveAsync` can both be active concurrently — `ClientWebSocket` supports one concurrent send + one concurrent receive (but NOT multiple concurrent sends or multiple concurrent receives). Our design uses exactly one of each, so this is safe.
## Error handling
| Failure mode | Behavior | Exit |
|-|-|-|
| `~/.openai_key` missing on Pi | `bin/probe-cs-4a` precondition check exits before publishing. | 1 |
| `~/.openai_key` exists but empty / not `sk-`-shaped | `LoadApiKey()` logs `[error] API key file empty` / `[error] API key doesn't start with sk-`, exits. | 2 |
| `ws.ConnectAsync` throws (DNS, TLS, 401, 403, 429, 500, network down) | Log `[error] WS connect failed: {ex.Message}` plus inner `HttpRequestException.StatusCode` if present. | 3 |
| Server `error` event mid-session | ReceiveLoop logs the full event JSON, `cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`; `finally` runs CloseAsync (no-throw with timeout) + stream.Stop + PortAudio.Terminate. | 4 |
| WS drops mid-conversation (`WebSocketException` in ReceiveLoop) | Same as above — log + cancel + cleanup-via-finally. | 4 |
| PortAudio input stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 5 |
| PortAudio output stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 6 |
| Server takes forever and never sends `response.done` | No timeout in 4a. User's recourse is Ctrl-C. (If verification surfaces this, add a configurable timeout in 4b — out of scope for 4a.) | 0 (via Ctrl-C) |
| Mic callback can't `TryWrite` to `upstreamChannel` (UpstreamSendLoop falling behind) | Log `[status] upstream behind, dropping mic frame`, continue. (Same shape as Test 3's "consumer behind" log.) | 0 |
| Workstation Ctrl-C | `Console.CancelKeyPress``e.Cancel = true; cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`. `finally` runs full cleanup path. | 0 |
The `finally` block at the bottom of `Program.cs`'s outer `try` runs **once** and is the single point where WS is closed, streams are stopped, and PortAudio is terminated. Cleanup is idempotent (`stream.Stop` is a no-op if already stopped; `PortAudio.Terminate` is fine to call once even if some path didn't run).
## Gotchas carried forward (from findings.md)
1. **`Libc.setenv("PA_ALSA_PLUGHW", "1", 1)`** before any PortAudio init. `Environment.SetEnvironmentVariable` alone does NOT propagate to libc on Linux. Apply alongside the managed call.
2. **No OrtEnv work in 4a** — no `Microsoft.ML.OnnxRuntime` dependency. (4b reintroduces it.)
3. **`using Stream = PortAudioSharp.Stream;`** at the top of `Program.cs` to disambiguate from `System.IO.Stream` that `ImplicitUsings` pulls in.
4. **`<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`** in `Probe.csproj` for the input/output callbacks' `Buffer.MemoryCopy` over `short*`.
5. **Fire-and-forget output stream lifetime.** `stream.Start()` runs synchronously under try/catch BEFORE any auxiliary cleanup task launches; on `Start()` failure, dispose + rethrow without ever leaking the stream. Test 4a uses a *persistent* output stream (opened once, drained for both the beep and the response audio, stopped on shutdown), so there's no per-cycle cleanup task — the `Start()` failure path is the only one that matters.
6. **Streaming-length cleanup discipline.** Test 3's "scale `done.Wait(timeout)` to recordBuf length" trick does NOT apply because the assistant audio length is unknown a priori. Replaced by the output callback's `_currentChunk == null && Reader.Count == 0 && Reader.Completion.IsCompleted` triple check. The `ManualResetEventSlim doneFlag` carries the signal to the main thread.
7. **`ssh -t` in `bin/probe-cs-4a`** for Ctrl-C propagation from workstation SIGINT.
8. **Always wipe-and-replace the whole publish directory.** A partial `scp Probe` after a rebuild leaves the old `.dll` on the target, which the new launcher loads. `bin/probe-cs-4a` does `ssh "rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a"` BEFORE the `scp -r` (Test 3 pattern).
9. **Startup beep must not be captured by the mic and shipped to OpenAI.** The `_micArmed` flag stays `false` until ~300 ms after the beep is queued to `downstreamChannel` (200 ms beep + 100 ms drain margin). The mic callback discards frames while `!_micArmed`.
New for 4a (none carried forward from prior tests — these are surfaced here for the first time):
10. **`ClientWebSocket` concurrency rule:** one concurrent Send + one concurrent Receive is supported; multiple concurrent Sends (or Receives) is not. Our design has exactly one SendLoop and one ReceiveLoop, so this is safe by construction — but if a future change adds a second sender (e.g. a ping task), it must serialize with the existing one.
11. **Channel completion semantics:** `Writer.Complete()` makes `Reader.Completion.IsCompleted` true ONLY after all written items are consumed. The output callback's triple check accounts for this correctly. Don't change to a different completion sentinel without re-verifying the corner case.
12. **API key file ownership:** the precondition check in `bin/probe-cs-4a` verifies *presence* and *readability* by `pi`, not strict mode 600. The probe itself logs a non-fatal warning if the mode is looser than 600; this lets a slightly looser local setup proceed.
## Verification (hardware only, no automated tests)
One-time setup on the Pi (workstation tells you to do this; `bin/probe-cs-4a` precondition-checks it before publishing):
```sh
sshpass -p 'assistant' ssh pi@192.168.50.115 \
'umask 077 && printf "%s\n" "sk-PASTE-YOUR-KEY-HERE" > ~/.openai_key'
```
Run from workstation:
```sh
./bin/probe-cs-4a
```
Expected console flow (abridged):
```
>> precondition: ~/.openai_key present on Pi. OK.
>> dotnet publish (linux-arm64, self-contained)
>> scp to pi@192.168.50.115:~/probe-cs-4a/ (wipe-and-replace)
>> ssh + run on Pi (Ctrl-C from this terminal stops the probe)
[ws] connecting to wss://api.openai.com/v1/realtime?model=gpt-realtime
[ws] connected
[ws] session.created sess_xxxxxxxxxxxx
[ws] session.updated — session pinned
Using device 3 ('USB Speaker Phone (LISTENAI ...)')
[beep]
Speak now.
[vad] speech started
[vad] speech stopped
[ws] response.created
[audio] first delta received (t = … s since speech_stopped)
[audio] playback drained
[transcript] Hello! How can I help you today?
[ws] closing
bye
```
### Pass criteria
All must hold during a single run on the Pi:
1. **End-to-end round-trip works once.** Probe boots, beeps, you speak a short sentence, the assistant's spoken reply plays back recognisably through the same USB Speaker Phone, the probe exits 0.
2. **Response is intelligible.** The assistant's audio is clear speech, no clipping, no dropouts longer than a single 43 ms callback (one zero-fill is fine and expected on the first chunk arrival).
3. **Round-trip latency.** Time from `[vad] speech stopped` log to `[audio] first delta received` log (i.e. first response audio byte *arriving* at the probe — actual speaker emission adds a fixed ~43 ms output-callback period on top) is **under ~2 s**. Slower is acceptable if explicitly logged; **over ~5 s is a fail.**
4. **No `[status] upstream behind` warnings** during normal operation.
5. **No `[error] WS …` events** during normal operation.
6. **No PortAudio `InputOverflow` status flags** during normal operation.
7. **Workstation Ctrl-C** at any point produces a clean exit: WS closes with normal closure, no stack trace, terminal returns to prompt. Process exit code 0 on Ctrl-C during steady listen; non-zero on Ctrl-C during a failure path is fine.
8. **`response.done`-followed-cleanup signaling works.** Probe exits within ~500 ms of the last audio sample being heard. No long hang.
### What success teaches us for 4b
- Auth + WS handshake + headers + URL are correct.
- `session.update` with `server_vad` works; the server fires `speech_started` / `speech_stopped` correctly on real Pi mic input.
- Base64 decode of `response.audio.delta` produces playable PCM16 at 24 kHz on the USB Speaker Phone (PortAudio plug-resamples 48↔24 at ALSA without trouble).
- The streaming-length cleanup discipline (`_noMoreDeltas` + `Channel.Completion.IsCompleted` + working-chunk check) is correct.
- The `Channel<short[]>` upstream/downstream pattern handles backpressure cleanly.
- `bin/probe-cs-4a`'s `~/.openai_key` precondition pattern is the right key-management story (it'll lift verbatim into 4b and the main assistant).
### What failure would tell us
- **401 / 403 on connect:** API key is wrong or the file's been mangled.
- **WS connects but `session.updated` never arrives:** the session config is malformed for the current Realtime API surface — `modalities`, `turn_detection`, or `voice` is using an outdated field name or value.
- **No `speech_stopped` event ever fires:** `turn_detection` config is wrong for the current API (field renamed?), or the mic is silent (gain too low, wrong device, ALSA plug-rate-conversion broken at 24 kHz). Cross-check by adding a periodic RMS log in the mic callback.
- **`[status] upstream behind` repeats:** UpstreamSendLoop is slower than the mic callback. Most likely `ws.SendAsync` is blocking on TCP. Cheap remediation: bump channel capacity. Real remediation: investigate the WS backpressure.
- **First delta takes >5 s:** server-side issue OR our `instructions` is producing a long thinking pause. Tighten instructions or report and move on.
- **Garbled / chipmunk-speed playback:** sample-rate mismatch. Verify the output stream actually opened at 24 kHz (PortAudio might have silently substituted a default).
- **Click at the start of playback:** zero-fill underrun ran before the first delta. Acceptable for a probe; if it bothers you, add a small pre-roll buffer.
- **Probe hangs on exit:** `Writer.Complete()` was never called (receive loop missed `response.done`), or the output callback's triple check is wrong. Investigate via logging.
## Reference paths
- Spec (this doc): `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`
- Plan (next, to write): `docs/superpowers/plans/2026-06-12-test-4a-openai-realtime.md`
- Code (to be written): `tests/04a-realtime-oneshot-cs/`
- Deploy (to be written): `bin/probe-cs-4a`
- Test 3 reference: `tests/03-full-cycle-cs/Program.cs`, `bin/probe-cs-3`, `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
- API key file location: `/home/pi/.openai_key` on the Pi, mode 600, single line of `sk-…`
- Canonical findings: `findings.md` § "C# probe outcome" through § "C# full-cycle outcome"
- OpenAI Realtime API surface as of this brainstorm:
- WebSocket URL: `wss://api.openai.com/v1/realtime?model=gpt-realtime`
- Headers: `Authorization: Bearer <key>`, `OpenAI-Beta: realtime=v1`
- Audio formats: `pcm16` (default 24 kHz Int16), `g711_ulaw`, `g711_alaw`
- Session config keys used: `modalities`, `instructions`, `voice`, `input_audio_format`, `output_audio_format`, `turn_detection`
- `turn_detection.server_vad` subkeys used: `threshold`, `prefix_padding_ms`, `silence_duration_ms`, `create_response`
- Server events consumed: `session.created`, `session.updated`, `input_audio_buffer.speech_started`, `input_audio_buffer.speech_stopped`, `input_audio_buffer.committed`, `response.created`, `response.audio_transcript.delta`, `response.audio.delta`, `response.audio.done`, `response.done`, `error`
+69
View File
@@ -244,6 +244,75 @@ Test 3 stitched the C# Test 1 (record + play) and Test 2 (wakeword + beep) into
- Code: `tests/03-full-cycle-cs/`
- Deploy: `bin/probe-cs-3`
## Test 4a outcome (2026-06-12 — OpenAI Realtime one-shot voice round-trip in C#)
Test 4a opens a WebSocket to `wss://api.openai.com/v1/realtime?model=gpt-realtime`, plays a startup beep, streams mic audio to the server VAD endpoint, plays the assistant's audio reply back through the same USB Speaker Phone, and exits. Probe lives at `tests/04a-realtime-oneshot-cs/`; deploy script `bin/probe-cs-4a`. All 8 pass criteria met on hardware after a four-round iteration to match the GA Realtime API surface (the spec was written against beta-era docs from openai/openai-realtime-api-beta).
### What we proved
- **`ClientWebSocket` + two background `Task.Run` loops (ReceiveLoop, UpstreamSendLoop) + bounded `Channel<short[]>` on each direction** is the right shape on .NET 9 / linux-arm64 for a Realtime API client. No third-party WebSocket lib needed; `System.Text.Json` is fine for event parsing via `JsonDocument`.
- **One PortAudio input stream + one persistent output stream at 24 kHz mono Int16, both on the USB Speaker Phone, coexist cleanly.** PortAudio plug-resamples 48↔24 at ALSA on this device. No client-side resampling.
- **Server VAD endpointing works out of the box.** `turn_detection: { type: "server_vad", threshold: 0.5, prefix_padding_ms: 300, silence_duration_ms: 500, create_response: true }` (nested under `session.audio.input`) fires `speech_started` / `speech_stopped` correctly on real Pi mic input; server auto-commits the buffer and creates the response.
- **Round-trip latency is excellent.** End-of-speech (server VAD `speech_stopped`) to first response audio sample arriving at the probe: **0.44 s** measured on the test "hello" turn. Spec target was under 2 s; we beat it by 4×.
- **Streaming-length cleanup discipline works.** Output callback signals `doneFlag` when `CurrentChunk == null && Reader.Count == 0 && NoMoreDeltas` (the `NoMoreDeltas` `volatile bool` is set on `response.done` after `Writer.Complete()`; reading it is cheaper than touching `Reader.Completion`). Probe exits ~immediately after `response.done`.
- **`~/.openai_key` precondition pattern in `bin/probe-cs-4a` works.** Pre-publish check via `sshpass ... 'test -f ~/.openai_key && test -r ~/.openai_key'` catches a missing key at the deploy step rather than 30 s into publish + scp. Set with `umask 077 && cat > ~/.openai_key <<KEY` on the Pi.
### Key gotchas — Realtime API GA migration
The spec was written against the older beta API surface (the openai/openai-realtime-api-beta library docs, which were the largest indexed source). The GA API rejected every beta-era piece of our session.update payload, surfacing one error per iteration. The full diff (commit `4cfb0d4`):
1. **No `OpenAI-Beta: realtime=v1` request header.** GA rejects it as `beta_api_shape_disabled`. Only `Authorization: Bearer <key>` is required for GA.
2. **`session.update` payload is nested under `session.audio.{input,output}` instead of flat at the top of `session`.** Beta had `session.input_audio_format`, `session.output_audio_format`, `session.voice`, `session.turn_detection`. GA has `session.audio.input.format`, `session.audio.input.turn_detection`, `session.audio.output.format`, `session.audio.output.voice`. Also `session.type = "realtime"` and `session.model = "gpt-realtime"` go inline at the top of `session`.
3. **`session.audio.{input,output}.format` is now an OBJECT, not a bare string.** Beta accepted `"pcm16"`. GA requires `{ "type": "audio/pcm", "rate": 24000 }`. The error was `Invalid type for 'session.audio.input.format': expected an object, but got a string instead.`
4. **`session.modalities` is no longer accepted.** Server rejected with `Unknown parameter: 'session.modalities'`. Dropped from our payload; server defaults are fine (audio + text output). The GA equivalent for constraining output is likely `output_modalities` (matches Responses API convention) — not needed for the probe.
5. **Server events for response audio are renamed `response.audio.*` → `response.output_audio.*`.** Beta had `response.audio.delta` / `.done` / `response.audio_transcript.delta` / `.done`. GA has `response.output_audio.delta` / `.done` / `response.output_audio_transcript.delta` / `.done`. **This is the silent failure mode**: the receive loop's switch default branch quietly logs `[ws] ignored type=…` instead of writing the decoded audio to the output channel. The probe logs "session pinned", VAD fires, response completes, transcript is empty, no audio is heard. Took one round of hardware verification to surface; trivially diagnosed via the `ignored type=…` log lines.
6. **GA emits a stream of conversation/response framing events** that don't exist in beta: `conversation.item.added` / `.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `rate_limits.updated`. Logged as `[ws] ignored type=…` initially; now silenced in our switch so they don't bury real signal.
The current Realtime model name `gpt-realtime` is still accepted (the dev docs mention `gpt-realtime-2` as well; we didn't need to swap).
### Other gotchas
- **Mid-startup WS error must route through cleanup.** Initial implementation had a single `try { … } finally { … }` around main. When the receive loop processed an error event and called `cts.Cancel()` mid-`SendSessionUpdate`, the `await ws.SendAsync(…, cts.Token)` threw `TaskCanceledException`, which propagated past the finally as an unhandled exception (PortAudio + ws Dispose still ran from finally, but the process exited non-zero with a stack trace). Fix: add `catch (OperationCanceledException) { /* expected on cancel */ } catch (Exception ex) { log; ExitCode = 7 }` between the outer try and finally. Probe now logs the `[error] WS error event:` from the receive loop and exits cleanly. Commit `4cfb0d4`.
- **`Environment.Exit(N)` skips the outer `finally` and leaks PortAudio**, but is OK in the LoadApiKey path because it runs before `PortAudio.Initialize()`. The `Environment.ExitCode = N; return;` pattern in WS-connect / stream-Start failure paths preserves the finally correctly.
- **CA1416 warning on `File.GetUnixFileMode`** is benign; the call is guarded by `try { … } catch (PlatformNotSupportedException) { }`. The build emits one warning consistently; no action needed.
- **ALSA shutdown noise on first run.** Two `alsa_snd_pcm_drop` "failed" lines appeared from PortAudio's cleanup on the *failed* runs (cts.Cancel from an error event tore down both streams mid-init). Did NOT appear on the successful run. Looks like a PortAudio cleanup-on-abort quirk; only present on error paths.
### Pass criteria
| # | Criterion | Result |
|-|-|-|
| 1 | End-to-end round-trip works once | ✅ Probe boots, beeps, user speaks, audible reply, exits 0 |
| 2 | Response intelligible | ✅ Clean speech, no clipping, no dropouts |
| 3 | Latency < 2 s | ✅ **0.44 s** end-of-speech to first delta |
| 4 | No `[status] upstream behind` | ✅ None |
| 5 | No `[error] WS …` during normal flow | ✅ None |
| 6 | No PortAudio `InputOverflow` | ✅ None |
| 7 | Ctrl-C clean exit | ✅ Same cleanup path as the successful run, which exited cleanly |
| 8 | Exits within ~500 ms of last audio sample | ✅ `response.done` → `[ws] closing` → `bye` is instant |
### Implications for Test 4b
- **GA API surface is now proven.** Test 4b inherits the migrated `SendSessionUpdate` payload, the renamed receive-loop switch cases, the silenced GA framing events, and the outer try/catch shape. Lift them directly.
- **The "spec written against beta docs" failure mode is a real risk for any future Realtime API work.** When LLM-fetched docs (context7, training data) disagree with the live server, **only the server is authoritative**. Pattern: send a minimal session.update, observe the error response, iterate. Our four-round iteration took ~10 min total.
- **The "[ws] ignored type=…" log line on the default branch is the diagnostic** that surfaced the renames. Keep it (or an equivalent) in 4b's receive loop. Without it, missing audio could have been silently attributed to a thousand other causes.
- **Server VAD parameters (`threshold: 0.5`, `silence_duration_ms: 500`, `prefix_padding_ms: 300`) work well on this hardware** — natural-feeling end-of-turn after ~500 ms of silence, no clipped openings of user speech. Lift unchanged into 4b.
- **`Environment.Exit` vs `Environment.ExitCode = N; return;`** — the latter is the correct pattern for failure paths that need PortAudio + WS cleanup. 4b will have more failure paths (wakeword model init, audio rate conversion, etc.); use the ExitCode pattern consistently.
- **The startup beep pattern (queue beep onto the persistent output channel before `_micArmed = true`)** works — beep does not get captured and shipped to OpenAI. Carry into 4b.
- **The 16↔24 kHz coexistence question is the one major thing 4a did not address.** 4a is 24 kHz end-to-end. 4b must deal with the wakeword model requiring 16 kHz input while OpenAI requires 24 kHz pcm16 — either a second input stream, an in-process resampler, or open one stream at 24 kHz and downsample for the wakeword. The spec/plan for 4b will need to pick.
### Reference paths (Test 4a)
- Spec: `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`
- Plan: `docs/superpowers/plans/2026-06-12-test-4a-openai-realtime.md`
- Code: `tests/04a-realtime-oneshot-cs/`
- Deploy: `bin/probe-cs-4a`
- GA Realtime API headers / URL / session.update shape (as discovered against the live server, 2026-06-12):
- `wss://api.openai.com/v1/realtime?model=gpt-realtime`
- Header `Authorization: Bearer <key>` (NO `OpenAI-Beta`)
- `session.update` payload: `{ type: "session.update", session: { type: "realtime", model: "gpt-realtime", instructions: "...", audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: { type: "server_vad", threshold, prefix_padding_ms, silence_duration_ms, create_response } }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "alloy" } } } }`
- Server events to handle: `session.created`, `session.updated`, `input_audio_buffer.speech_started`, `input_audio_buffer.speech_stopped`, `input_audio_buffer.committed`, `response.created`, `response.output_audio_transcript.delta`, `response.output_audio.delta`, `response.output_audio.done`, `response.output_audio_transcript.done`, `response.done`, `error`
- Server events safe to ignore for a probe: `conversation.item.added`, `conversation.item.done`, `response.output_item.added`, `response.output_item.done`, `response.content_part.added`, `response.content_part.done`, `rate_limits.updated`
## Reference paths
- Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
@@ -0,0 +1,18 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>RealtimeOneShotProbe</RootNamespace>
<AssemblyName>Probe</AssemblyName>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<PublishSingleFile>false</PublishSingleFile>
<InvariantGlobalization>true</InvariantGlobalization>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PortAudioSharp2" Version="1.0.6" />
</ItemGroup>
</Project>
+542
View File
@@ -0,0 +1,542 @@
using System.Net.WebSockets;
using System.Runtime.InteropServices;
using System.Text;
using System.Text.Json;
using System.Threading.Channels;
using PortAudioSharp;
using Stream = PortAudioSharp.Stream;
// .NET-side mirror for any readers that go through Environment.GetEnvironmentVariable,
// AND libc setenv so PortAudio (which uses getenv()) sees the value.
Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1");
Libc.setenv("PA_ALSA_PLUGHW", "1", 1);
string apiKey = LoadApiKey();
PortAudio.Initialize();
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var upstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.UpstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = true,
SingleReader = true,
});
var downstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(C.DownstreamChannelCapacity)
{
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = false,
SingleReader = true,
});
var doneFlag = new ManualResetEventSlim(false);
var state = new SharedState();
var ws = new ClientWebSocket();
ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
// No OpenAI-Beta header — the GA Realtime API rejects beta-shaped requests with
// "beta_api_shape_disabled". Documented at developers.openai.com/api/docs/guides/realtime-websocket.
Stream? inputStream = null;
Stream? outputStream = null;
Task? receiveTask = null;
Task? sendTask = null;
try
{
var uri = new Uri($"wss://api.openai.com/v1/realtime?model={C.ModelName}");
Console.WriteLine($"[ws] connecting to {uri}");
try
{
await ws.ConnectAsync(uri, cts.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] WS connect failed: {ex.Message}");
if (ex.InnerException is HttpRequestException httpEx)
Console.Error.WriteLine($"[error] inner: HTTP {(int?)httpEx.StatusCode}");
Environment.ExitCode = 3;
return;
}
Console.WriteLine("[ws] connected");
receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel, cts, state));
await SendSessionUpdate(ws, cts.Token);
int device = FindUsbDevice();
Console.WriteLine($"Using device {device} ('{PortAudio.GetDeviceInfo(device).name}')");
inputStream = OpenInputStream(device, upstreamChannel, state);
try { inputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] input stream start failed: {ex.Message}");
Environment.ExitCode = 5;
return;
}
outputStream = OpenOutputStream(device, downstreamChannel, state, doneFlag);
try { outputStream.Start(); }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] output stream start failed: {ex.Message}");
Environment.ExitCode = 6;
return;
}
sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel, cts, state));
// Startup beep — queued onto downstreamChannel before the receive loop produces any audio.
short[] beep = MakeBeep(C.BeepHz, C.BeepSeconds, C.SampleRate);
Console.WriteLine("[beep]");
await downstreamChannel.Writer.WriteAsync(beep, cts.Token);
// Wait for beep to drain (200 ms beep + 100 ms margin) so it doesn't get captured
// by the mic and shipped to OpenAI.
Thread.Sleep((int)(C.BeepSeconds * 1000) + C.BeepDrainMs);
state.MicArmed = true;
Console.WriteLine("Speak now.");
try
{
doneFlag.Wait(cts.Token);
}
catch (OperationCanceledException) { /* Ctrl-C path */ }
Console.WriteLine("[ws] closing");
}
catch (OperationCanceledException)
{
// Mid-startup cancellation (typically a WS error event fired cts.Cancel while
// the main thread was awaiting SendSessionUpdate / a channel write).
// The receive loop has already logged the underlying [error]; we just route through cleanup.
}
catch (Exception ex)
{
Console.Error.WriteLine($"[error] unexpected: {ex.GetType().Name}: {ex.Message}");
Environment.ExitCode = 7;
}
finally
{
// Cancel any background work first so receive/send loops exit cleanly.
cts.Cancel();
if (ws.State == WebSocketState.Open || ws.State == WebSocketState.CloseReceived)
{
try
{
using var closeTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(2));
await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", closeTimeout.Token);
}
catch (Exception ex)
{
Console.Error.WriteLine($"[ws] close failed: {ex.Message}");
}
}
try { inputStream?.Stop(); } catch { }
try { outputStream?.Stop(); } catch { }
try { inputStream?.Dispose(); } catch { }
try { outputStream?.Dispose(); } catch { }
try { ws.Dispose(); } catch { }
try { PortAudio.Terminate(); } catch { }
Console.WriteLine("bye");
}
static string LoadApiKey()
{
string home = Environment.GetEnvironmentVariable("HOME") ?? "/root";
string path = Path.Combine(home, ".openai_key");
if (!File.Exists(path))
{
Console.Error.WriteLine($"[error] API key file not found at {path}");
Environment.Exit(2);
}
// Non-fatal mode-600 check.
try
{
var mode = File.GetUnixFileMode(path);
var leaky = mode & (UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute
| UnixFileMode.OtherRead | UnixFileMode.OtherWrite | UnixFileMode.OtherExecute);
if (leaky != UnixFileMode.None)
Console.Error.WriteLine($"[warn] {path} mode includes group/other perms ({mode}); expected 600");
}
catch (PlatformNotSupportedException) { /* not Unix */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[warn] could not check {path} mode: {ex.Message}");
}
string key = File.ReadAllText(path).Trim();
if (string.IsNullOrEmpty(key))
{
Console.Error.WriteLine($"[error] API key file empty");
Environment.Exit(2);
}
if (!key.StartsWith("sk-"))
{
Console.Error.WriteLine($"[error] API key doesn't start with sk-");
Environment.Exit(2);
}
return key;
}
static int FindUsbDevice()
{
int count = PortAudio.DeviceCount;
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
if (info.name.ToLowerInvariant().Contains("usb") && info.maxInputChannels >= 1)
return i;
}
Console.Error.WriteLine("USB audio device not found. Devices:");
for (int i = 0; i < count; i++)
{
var info = PortAudio.GetDeviceInfo(i);
Console.Error.WriteLine(
$" [{i}] {info.name} in={info.maxInputChannels} out={info.maxOutputChannels}");
}
Environment.Exit(1);
return -1; // unreachable
}
static short[] MakeBeep(double freqHz, double durationS, int sampleRate)
{
int n = (int)(sampleRate * durationS);
var buf = new short[n];
for (int i = 0; i < n; i++)
{
double t = i / (double)sampleRate;
double v = 0.3 * Math.Sin(2.0 * Math.PI * freqHz * t);
buf[i] = (short)(v * short.MaxValue);
}
return buf;
}
static async Task SendSessionUpdate(ClientWebSocket ws, CancellationToken ct)
{
// GA Realtime API shape: nested session.audio.{input,output} with format + voice + turn_detection
// moved under audio.input. session.type = "realtime" and session.model inline.
// (Beta-era flat shape with session.input_audio_format/turn_detection at the top is rejected.)
var msg = new
{
type = "session.update",
session = new
{
type = "realtime",
model = C.ModelName,
instructions = C.Instructions,
// No 'modalities' field — the beta name. GA defaults are fine; if we ever need to
// constrain to audio-only we'd use 'output_modalities' (the documented GA name in
// newer OpenAI APIs).
audio = new
{
input = new
{
// GA shape: format is now an object with a "type" field, not a bare string.
format = new { type = "audio/pcm", rate = 24_000 },
turn_detection = new
{
type = "server_vad",
threshold = 0.5,
prefix_padding_ms = 300,
silence_duration_ms = 500,
create_response = true,
},
},
output = new
{
format = new { type = "audio/pcm", rate = 24_000 },
voice = C.Voice,
},
},
},
};
byte[] bytes = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(bytes), WebSocketMessageType.Text, endOfMessage: true, ct);
}
static Stream OpenInputStream(int device, Channel<short[]> upstream, SharedState state)
{
var inParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr input, IntPtr _, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags status, IntPtr _3) =>
{
if (status.HasFlag(StreamCallbackFlags.InputOverflow))
Console.Error.WriteLine("[status] input overflow");
if (!state.MicArmed || state.StopSending)
return StreamCallbackResult.Continue;
if (frameCount != C.MicBlockFrames)
{
Console.Error.WriteLine($"[status] unexpected input frameCount={frameCount} (want {C.MicBlockFrames})");
return StreamCallbackResult.Continue;
}
var frame = new short[C.MicBlockFrames];
unsafe
{
short* src = (short*)input.ToPointer();
fixed (short* dst = &frame[0])
Buffer.MemoryCopy(src, dst, C.MicBlockFrames * sizeof(short), C.MicBlockFrames * sizeof(short));
}
if (!upstream.Writer.TryWrite(frame))
Console.Error.WriteLine("[status] upstream behind, dropping mic frame");
return StreamCallbackResult.Continue;
};
return new Stream(inParams, null, C.SampleRate, C.MicBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static Stream OpenOutputStream(int device, Channel<short[]> downstream, SharedState state, ManualResetEventSlim doneFlag)
{
var outParams = new StreamParameters
{
device = device,
channelCount = C.Channels,
sampleFormat = SampleFormat.Int16,
suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowOutputLatency,
hostApiSpecificStreamInfo = IntPtr.Zero,
};
Stream.Callback callback = (IntPtr _, IntPtr output, uint frameCount,
ref StreamCallbackTimeInfo _2, StreamCallbackFlags _3, IntPtr _4) =>
{
int fill = 0;
while (fill < frameCount)
{
// Pull next chunk if we have no working one.
if (state.CurrentChunk == null)
{
if (!downstream.Reader.TryRead(out var nextChunk))
break; // ring empty
state.CurrentChunk = nextChunk;
state.ChunkOffset = 0;
}
// Local non-null alias so the compiler doesn't warn on dereference inside `fixed`.
short[] chunk = state.CurrentChunk!;
int slack = (int)frameCount - fill;
int remaining = chunk.Length - state.ChunkOffset;
int take = Math.Min(remaining, slack);
unsafe
{
short* dst = (short*)output.ToPointer();
fixed (short* src = &chunk[state.ChunkOffset])
Buffer.MemoryCopy(src, dst + fill, take * sizeof(short), take * sizeof(short));
}
state.ChunkOffset += take;
fill += take;
if (state.ChunkOffset >= chunk.Length)
state.CurrentChunk = null;
}
unsafe
{
short* dst = (short*)output.ToPointer();
for (int i = fill; i < (int)frameCount; i++) dst[i] = 0;
}
// Completion = working chunk drained AND channel empty AND no more deltas expected.
// _noMoreDeltas is the spec's "redundant fast path" — set immediately on response.done,
// avoids touching downstream.Reader.Completion (a Task allocation per callback).
if (state.CurrentChunk == null
&& downstream.Reader.Count == 0
&& state.NoMoreDeltas)
{
doneFlag.Set();
return StreamCallbackResult.Complete;
}
return StreamCallbackResult.Continue;
};
return new Stream(null, outParams, C.SampleRate, C.OutBlockFrames, StreamFlags.NoFlag, callback, IntPtr.Zero);
}
static async Task UpstreamSendLoop(ClientWebSocket ws, Channel<short[]> upstream, CancellationTokenSource cts, SharedState state)
{
try
{
await foreach (var frame in upstream.Reader.ReadAllAsync(cts.Token))
{
if (state.StopSending)
{
// After server VAD said the user is done, drop in-flight mic frames and exit
// once the channel is drained. Matches the spec's "_stopSending && empty" exit.
if (upstream.Reader.Count == 0) break;
continue;
}
ReadOnlySpan<byte> bytes = MemoryMarshal.AsBytes(frame.AsSpan());
string base64 = Convert.ToBase64String(bytes);
var msg = new { type = "input_audio_buffer.append", audio = base64 };
byte[] json = JsonSerializer.SerializeToUtf8Bytes(msg);
await ws.SendAsync(new ArraySegment<byte>(json), WebSocketMessageType.Text, endOfMessage: true, cts.Token);
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] upstream send: {ex.Message}");
cts.Cancel();
}
}
static async Task ReceiveLoop(ClientWebSocket ws, Channel<short[]> downstream, CancellationTokenSource cts, SharedState state)
{
var transcript = new StringBuilder();
var buffer = new byte[4096];
bool firstDeltaSeen = false;
DateTime speechStoppedTime = DateTime.MinValue;
using var msgBuffer = new MemoryStream();
try
{
while (!cts.IsCancellationRequested)
{
msgBuffer.SetLength(0);
WebSocketReceiveResult result;
do
{
result = await ws.ReceiveAsync(new ArraySegment<byte>(buffer), cts.Token);
if (result.MessageType == WebSocketMessageType.Close) return;
msgBuffer.Write(buffer, 0, result.Count);
} while (!result.EndOfMessage);
msgBuffer.Position = 0;
using var doc = JsonDocument.Parse(msgBuffer);
string type = doc.RootElement.GetProperty("type").GetString() ?? "";
switch (type)
{
case "session.created":
{
string? sid = doc.RootElement.TryGetProperty("session", out var sess)
&& sess.TryGetProperty("id", out var idEl) ? idEl.GetString() : null;
Console.WriteLine($"[ws] session.created {sid}");
break;
}
case "session.updated":
Console.WriteLine("[ws] session.updated — session pinned");
break;
case "input_audio_buffer.speech_started":
Console.WriteLine("[vad] speech started");
break;
case "input_audio_buffer.speech_stopped":
Console.WriteLine("[vad] speech stopped");
speechStoppedTime = DateTime.UtcNow;
state.StopSending = true;
break;
case "input_audio_buffer.committed":
Console.WriteLine("[ws] input_audio_buffer.committed");
break;
case "response.created":
Console.WriteLine("[ws] response.created");
break;
// GA event names are response.output_audio* (beta was response.audio*).
case "response.output_audio_transcript.delta":
if (doc.RootElement.TryGetProperty("delta", out var tEl))
transcript.Append(tEl.GetString());
break;
case "response.output_audio.delta":
{
string b64 = doc.RootElement.GetProperty("delta").GetString() ?? "";
byte[] audioBytes = Convert.FromBase64String(b64);
short[] samples = new short[audioBytes.Length / 2];
Buffer.BlockCopy(audioBytes, 0, samples, 0, audioBytes.Length);
if (!firstDeltaSeen)
{
firstDeltaSeen = true;
double elapsed = speechStoppedTime == DateTime.MinValue
? -1
: (DateTime.UtcNow - speechStoppedTime).TotalSeconds;
Console.WriteLine($"[audio] first delta received (t = {elapsed:0.00} s since speech_stopped)");
}
await downstream.Writer.WriteAsync(samples, cts.Token);
break;
}
case "response.output_audio.done":
Console.WriteLine("[ws] response.output_audio.done");
break;
case "response.output_audio_transcript.done":
// transcript already accumulated; nothing to do here, server signals end of stream.
break;
// GA emits a stream of conversation/response framing events we don't need for the
// probe — quiet them so they don't bury real signal in the log.
case "conversation.item.added":
case "conversation.item.done":
case "response.output_item.added":
case "response.output_item.done":
case "response.content_part.added":
case "response.content_part.done":
case "rate_limits.updated":
break;
case "response.done":
state.NoMoreDeltas = true;
downstream.Writer.Complete();
Console.WriteLine("[ws] response.done");
Console.WriteLine($"[transcript] {transcript}");
break;
case "error":
Console.Error.WriteLine($"[error] WS error event: {doc.RootElement.GetRawText()}");
cts.Cancel();
break;
default:
Console.Error.WriteLine($"[ws] ignored type={type}");
break;
}
}
}
catch (OperationCanceledException) { /* expected on Cancel */ }
catch (Exception ex)
{
Console.Error.WriteLine($"[error] receive loop: {ex.Message}");
cts.Cancel();
}
}
sealed class SharedState
{
public volatile bool MicArmed;
public volatile bool StopSending;
public volatile bool NoMoreDeltas;
// Output-callback-thread-local; do NOT touch from anywhere else.
public short[]? CurrentChunk;
public int ChunkOffset;
}
static class C
{
public const int SampleRate = 24_000;
public const int Channels = 1;
public const uint MicBlockFrames = 1920; // 40 ms @ 24 kHz
public const uint OutBlockFrames = 1024; // ~43 ms @ 24 kHz
public const string ModelName = "gpt-realtime";
public const string Voice = "alloy";
public const string Instructions = "Reply in one short sentence.";
public const double BeepHz = 880.0;
public const double BeepSeconds = 0.2;
public const int BeepDrainMs = 100;
public const int UpstreamChannelCapacity = 64;
public const int DownstreamChannelCapacity = 64;
}
static class Libc
{
[DllImport("libc", EntryPoint = "setenv")]
public static extern int setenv(string name, string value, int overwrite);
}