tes
dea126f994
Boot → beep → user speaks → server VAD ends turn → assistant audio plays back → exit 0. No wakeword, no loop, no state machine. Splits Test 4 into 4a (this spec) and 4b (wakeword + loop + 16↔24 kHz coexistence, future). 4a debugs the OpenAI Realtime surface — auth, WS handshake, session.update payload, server VAD, base64 audio, streaming- length output-stream cleanup — in isolation from 4b's harder concurrency.
475 lines
36 KiB
Markdown
475 lines
36 KiB
Markdown
# Test 4a — OpenAI Realtime API one-shot voice round-trip (C#)
|
||
|
||
A throwaway C# probe that has a single round-trip spoken conversation with the OpenAI Realtime API:
|
||
|
||
> Boot → beep → user speaks one sentence → server VAD ends the turn → assistant audio reply plays back through the same USB Speaker Phone → exit 0.
|
||
|
||
No wakeword. No loop. No state machine. One process, runs once, dies.
|
||
|
||
Test 4a's job is to debug the OpenAI Realtime API surface (auth, WS handshake, `session.update` payload, event JSON shapes, base64 audio decode, server VAD endpointing, streaming-length output-stream cleanup) **in isolation** from the harder concurrency story of Test 4b (wakeword + loop + 16↔24 kHz coexistence).
|
||
|
||
Read `findings.md` (§ "C# probe outcome", § "C# wakeword probe outcome", § "C# full-cycle outcome") before reading this. Everything below assumes those outcomes as ground truth. The patterns lifted directly from Test 3 are: `Libc.setenv`, the `using Stream = PortAudioSharp.Stream;` import disambiguation, `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`, the fire-and-forget output stream lifetime (Start under try/catch BEFORE the cleanup task launches), `FindUsbDevice`, the `MakeBeep` helper, `bin/probe-cs-3`'s wipe-and-replace publish shape with `ssh -t`.
|
||
|
||
## Goal
|
||
|
||
A single-binary C# probe that:
|
||
|
||
1. Reads the OpenAI API key from `~/.openai_key` on the Pi (mode 600).
|
||
2. Connects to `wss://api.openai.com/v1/realtime?model=gpt-realtime` over a WebSocket with `Authorization: Bearer …` + `OpenAI-Beta: realtime=v1` headers.
|
||
3. Sends one `session.update` event pinning audio formats, voice, instructions, and `turn_detection: server_vad`.
|
||
4. Plays a startup beep through the USB Speaker Phone.
|
||
5. Opens a 24 kHz mono Int16 input stream on the same device; streams mic audio to OpenAI as `input_audio_buffer.append` events.
|
||
6. Waits for the server VAD to fire `input_audio_buffer.speech_stopped` (server then auto-commits the buffer and creates the response).
|
||
7. Stops sending mic audio at that point. Receives `response.audio.delta` chunks, base64-decodes them, plays them through a 24 kHz output stream on the same device.
|
||
8. On `response.done` AND the output buffer drained, closes the WebSocket cleanly, terminates PortAudio, exits 0.
|
||
9. Exits cleanly on Ctrl-C from the workstation (SIGINT propagated via `ssh -t`) at any stage.
|
||
|
||
## Non-goals
|
||
|
||
- **No wakeword.** No `WakewordModel.cs`. No 16 kHz audio path anywhere in 4a. (Test 4b layers that on top of this proven base.)
|
||
- **No state machine, no loop.** Probe runs once and exits.
|
||
- **No barge-in.** Once `speech_stopped` fires, mic frames are dropped on the floor. The user cannot interrupt the assistant in 4a.
|
||
- **No client-side resampling.** Mic = 24 kHz, output = 24 kHz, OpenAI = 24 kHz pcm16. PortAudio plug-resamples 48↔24 at ALSA on the USB device.
|
||
- **No WAV capture.** Logging is stdout/stderr only.
|
||
- **No retry on WS errors, no reconnect, no health check.** Fail loud, exit non-zero, log enough to diagnose.
|
||
- **No automated tests.** Hardware verification only.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────┐
|
||
│ ClientWebSocket │
|
||
│ wss://api.openai.com/v1/realtime? │
|
||
│ model=gpt-realtime │
|
||
│ Headers: Authorization: Bearer … │
|
||
│ OpenAI-Beta: realtime=v1 │
|
||
└────┬──────────────────────────┬──────┘
|
||
│ SendAsync │ ReceiveAsync
|
||
▼ ▼
|
||
┌──────────────────────────┐ ┌──────────────────────────────┐
|
||
│ UpstreamSendLoop │ │ ReceiveLoop (Task.Run) │
|
||
│ (Task.Run) │ │ while (!cts.Cancel) { │
|
||
│ drain upstreamChannel: │ │ msg = ReceiveAsync(…) │
|
||
│ pull short[1920] │ │ doc = JsonDocument.Parse │
|
||
│ → base64 │ │ switch (doc["type"]) │
|
||
│ → input_audio_ │ │ session.created → log │
|
||
│ buffer.append │ │ session.updated → log │
|
||
│ → ws.SendAsync │ │ speech_started → log │
|
||
│ exits when │ │ speech_stopped → │
|
||
│ _stopSending && empty│ │ _stopSending = true │
|
||
└──────────┬───────────────┘ │ response.audio.delta: │
|
||
│ (FullMode.Wait) │ base64 → short[] │
|
||
│ │ → downstreamChannel │
|
||
▲ │ .Writer.WriteAsync │
|
||
│ short[1920] │ audio_transcript.delta:│
|
||
│ (40 ms @ 24 kHz) │ append to log buf │
|
||
┌──────────┴───────────────┐ │ response.done → │
|
||
│ PortAudio input stream │ │ _noMoreDeltas = true │
|
||
│ 24 kHz mono Int16, │ │ downstreamChannel │
|
||
│ 1920 frames/cb │ │ .Writer.Complete │
|
||
│ callback: │ │ error → log + Cancel │
|
||
│ if !_micArmed: discard│ └────────────┬─────────────────┘
|
||
│ elif _stopSending: │ │
|
||
│ discard │ ▼
|
||
│ else: │ ┌──────────────────────────────┐
|
||
│ copy → short[] │ │ Channel<short[]> │
|
||
│ TryWrite to │ │ (bounded cap 64, │
|
||
│ upstreamChannel │ │ FullMode.Wait) │
|
||
└──────────────────────────┘ └────────────┬─────────────────┘
|
||
│
|
||
▼
|
||
┌──────────────────────────────────┐
|
||
│ PortAudio output stream │
|
||
│ 24 kHz mono Int16, │
|
||
│ 1024 frames/cb │
|
||
│ callback: │
|
||
│ fill = 0 │
|
||
│ while fill < frameCount: │
|
||
│ if currentChunk == null: │
|
||
│ try pull short[] or break │
|
||
│ take = min(remaining, slack)│
|
||
│ memcpy chunk → output │
|
||
│ zero-fill remaining │
|
||
│ if currentChunk == null │
|
||
│ && Reader.Count == 0 │
|
||
│ && Completion.IsCompleted: │
|
||
│ doneFlag.Set() │
|
||
│ return Complete │
|
||
│ return Continue │
|
||
└────────────┬─────────────────────┘
|
||
│
|
||
▼
|
||
main thread: doneFlag.Wait
|
||
→ ws.CloseAsync(NormalClosure)
|
||
→ input/output stream.Stop
|
||
→ PortAudio.Terminate
|
||
→ exit 0
|
||
```
|
||
|
||
One process. Three managed threads (main + UpstreamSendLoop + ReceiveLoop) plus PortAudio's input and output audio threads (managed by the library).
|
||
|
||
**Shared mutable state — the full list, small on purpose:**
|
||
|
||
- `Channel<short[]> upstreamChannel`, bounded 64, `FullMode.Wait`, single-writer (mic callback), single-reader (UpstreamSendLoop). Holds mic frames waiting to be base64'd and sent.
|
||
- `Channel<short[]> downstreamChannel`, bounded 64, `FullMode.Wait`, **multi-writer** (main thread writes the startup beep; ReceiveLoop writes response audio chunks; temporally separated but Channel<T>'s `SingleWriter` flag is a contract not a runtime check, so keep it `false`), single-reader (output callback). Holds decoded audio chunks waiting to be played. Its `Writer.Complete()` (called by ReceiveLoop on `response.done`) is the WS-stream-end signal for the output callback.
|
||
- `volatile bool _micArmed`: starts `false`; set `true` after the startup beep has been queued and drained. Mic frames before this point are discarded so the beep doesn't get captured and shipped to OpenAI.
|
||
- `volatile bool _stopSending`: set by ReceiveLoop on `input_audio_buffer.speech_stopped`. Read by both the mic callback (which then discards) and UpstreamSendLoop (which then drains its queue and exits).
|
||
- `volatile bool _noMoreDeltas`: set by ReceiveLoop on `response.done`. Not strictly required because `downstreamChannel.Writer.Complete()` carries the same signal — but kept as a redundant fast path that the output callback can check without touching the channel.
|
||
- `ManualResetEventSlim doneFlag`: set by the output callback when it has drained the channel after completion. Awaited by main.
|
||
- `CancellationTokenSource cts`: wired to `Console.CancelKeyPress` + any WS error. Cancels both Task.Run loops; main's `doneFlag.Wait(cts.Token)` honours it.
|
||
|
||
No locks. No `Interlocked`. The bounded `Channel<T>` handles all the cross-thread audio handoff with the right backpressure semantics.
|
||
|
||
## WebSocket lifecycle
|
||
|
||
Six discrete phases, all in `Program.cs` top-level statements + the two `Task.Run` loops:
|
||
|
||
1. **Load API key.** `LoadApiKey()` helper reads `~/.openai_key`, trims whitespace. Mode 600 verified — non-fatal warning if looser. Fatal error (exit 2) if the file is missing, empty, or doesn't start with `sk-`.
|
||
2. **Construct `ClientWebSocket`** with two request headers: `Authorization: Bearer <key>` and `OpenAI-Beta: realtime=v1`. `ConnectAsync` to `wss://api.openai.com/v1/realtime?model=gpt-realtime` with `cts.Token`. On failure (DNS, TLS, 401, 403, 429, 5xx), log the exception message + any HTTP status carried on `WebSocketException.WebSocketErrorCode` / inner `HttpRequestException.StatusCode`; exit 3.
|
||
3. **Launch ReceiveLoop** (`Task.Run`) before sending anything, so we don't miss `session.created`.
|
||
4. **Send `session.update`.** First client message; pins the session config (see "Client → server events" below). Don't strictly block on `session.updated` — the receive loop logs it asynchronously — but the spec expects it within ~200 ms.
|
||
5. **Open + start PortAudio streams.** Input stream first (24 kHz mono Int16, 1920 frames/cb), then output stream (24 kHz mono Int16, 1024 frames/cb). Both opened on `FindUsbDevice()`. Launch UpstreamSendLoop (`Task.Run`).
|
||
6. **Startup beep + arm mic.** Queue a 200 ms 880 Hz beep (`MakeBeep` lifted from Test 3, regenerated at 24 kHz) onto `downstreamChannel`. Sleep ~300 ms (beep + drain margin). Set `_micArmed = true`. Print `Speak now.` to stdout.
|
||
7. **Wait for `doneFlag`** on the main thread (with `cts.Token` for Ctrl-C). On wake-up: `ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", …)` with a 2 s timeout token. Stop input stream, stop output stream, terminate PortAudio, exit 0.
|
||
|
||
## Client → server event payloads
|
||
|
||
Three send-once messages on the main thread, plus a stream of `input_audio_buffer.append` from UpstreamSendLoop. All written with `JsonSerializer.Serialize` over `Dictionary<string, object?>` (lightest thing that doesn't require typed records for a probe).
|
||
|
||
```json
|
||
// (1) Pinned session config — first thing sent after WS connect.
|
||
{
|
||
"type": "session.update",
|
||
"session": {
|
||
"modalities": ["audio", "text"],
|
||
"instructions": "Reply in one short sentence.",
|
||
"voice": "alloy",
|
||
"input_audio_format": "pcm16",
|
||
"output_audio_format": "pcm16",
|
||
"turn_detection": {
|
||
"type": "server_vad",
|
||
"threshold": 0.5,
|
||
"prefix_padding_ms": 300,
|
||
"silence_duration_ms": 500,
|
||
"create_response": true
|
||
}
|
||
}
|
||
}
|
||
|
||
// (2) Per mic frame (~40 ms cadence), from UpstreamSendLoop:
|
||
{ "type": "input_audio_buffer.append", "audio": "<base64-pcm16-24k>" }
|
||
```
|
||
|
||
No `conversation.item.create` and no `response.create` from the client — server VAD's `create_response: true` makes the server commit and create the response automatically on `speech_stopped`.
|
||
|
||
`instructions: "Reply in one short sentence."` keeps the response audio bounded (~2-4 s of speech) so a probe run is fast to verify by ear and cheap on quota.
|
||
|
||
`voice: "alloy"` is the long-standing default voice and is documented as universally available for raw WS sessions.
|
||
|
||
## Server → client event handlers
|
||
|
||
ReceiveLoop uses a 4 KB receive buffer + standard `ReceiveAsync` loop with `EndOfMessage` accumulation into a `MemoryStream`. On message end, `JsonDocument.Parse`. Switch on the top-level `type` property's string value:
|
||
|
||
| `type` | Action |
|
||
|-|-|
|
||
| `session.created` | log `[ws] session.created <session_id>` |
|
||
| `session.updated` | log `[ws] session.updated — session pinned` |
|
||
| `input_audio_buffer.speech_started` | log `[vad] speech started` |
|
||
| `input_audio_buffer.speech_stopped` | log `[vad] speech stopped`; set `_stopSending = true` |
|
||
| `input_audio_buffer.committed` | log only — server confirms it sealed the buffer |
|
||
| `response.created` | log only |
|
||
| `response.audio_transcript.delta` | append `delta` to an accumulating `StringBuilder` |
|
||
| `response.audio.delta` | base64-decode `delta` → bytes → reinterpret as `short[]` → `await downstreamChannel.Writer.WriteAsync(samples, cts.Token)`. The `await` is the backpressure path: if the channel is full, this blocks the receive loop, which blocks the WS receive buffer — correct behavior for a slow speaker. |
|
||
| `response.audio.done` | log only |
|
||
| `response.done` | set `_noMoreDeltas = true`; `downstreamChannel.Writer.Complete()`; print the accumulated transcript |
|
||
| `error` | log the full event JSON; `cts.Cancel()` |
|
||
| anything else | log `[ws] ignored type=<…>` once (or at debug verbosity), continue |
|
||
|
||
## Audio buffering — input
|
||
|
||
- Mic callback runs every 40 ms (1920 samples @ 24 kHz mono Int16, matches `framesPerBuffer = 1920`).
|
||
- Callback allocates `short[1920]`, `Buffer.MemoryCopy`s the input pointer, `upstreamChannel.Writer.TryWrite(frame)`.
|
||
- On TryWrite failure (channel full = UpstreamSendLoop is behind): log `[status] upstream behind, dropping mic frame`, continue.
|
||
- Discard frames when `_micArmed == false` (during startup beep) or `_stopSending == true` (after `speech_stopped`).
|
||
- On the `InputOverflow` status flag from PortAudio: log `[status] input overflow`, continue.
|
||
|
||
UpstreamSendLoop pseudocode:
|
||
|
||
```
|
||
await ReceiveLoop signals session.updated (or just optimistically loop from start)
|
||
while (!cts.IsCancellationRequested):
|
||
if _stopSending && upstreamChannel.Reader.Count == 0: break
|
||
frame = await upstreamChannel.Reader.ReadAsync(cts.Token)
|
||
if _stopSending: continue // drop frames captured after server VAD cutoff
|
||
bytes = MemoryMarshal.AsBytes(frame.AsSpan())
|
||
base64 = Convert.ToBase64String(bytes)
|
||
json = JsonSerializer.SerializeToUtf8Bytes(new {
|
||
type = "input_audio_buffer.append",
|
||
audio = base64
|
||
})
|
||
await ws.SendAsync(json, WebSocketMessageType.Text, endOfMessage: true, cts.Token)
|
||
```
|
||
|
||
## Audio buffering — output
|
||
|
||
`downstreamChannel = Channel.CreateBounded<short[]>(new BoundedChannelOptions(64) { FullMode = BoundedChannelFullMode.Wait, SingleWriter = false, SingleReader = true })`. `SingleWriter = false` because the main thread queues the startup beep AND ReceiveLoop queues response audio chunks (temporally separated, but the flag is a hard contract).
|
||
|
||
Capacity rationale: 64 chunks × ~40-80 ms per `response.audio.delta` ≈ 2.5-5 s of buffered audio. Comfortable headroom against network jitter without holding silly amounts of memory.
|
||
|
||
ReceiveLoop writes `short[]` chunks of varying length (one per `response.audio.delta` event). On `response.done`, calls `downstreamChannel.Writer.Complete()`.
|
||
|
||
Output callback's job is to fill `frameCount` shorts every callback (1024 frames = ~43 ms @ 24 kHz). It maintains a `short[]? _currentChunk` + `int _chunkOffset` across calls. Pseudocode:
|
||
|
||
```
|
||
int fill = 0;
|
||
while (fill < frameCount) {
|
||
if (_currentChunk == null) {
|
||
if (!downstreamChannel.Reader.TryRead(out _currentChunk)) break; // drained
|
||
_chunkOffset = 0;
|
||
}
|
||
int slack = frameCount - fill;
|
||
int remaining = _currentChunk.Length - _chunkOffset;
|
||
int take = Math.Min(remaining, slack);
|
||
unsafe {
|
||
fixed (short* src = &_currentChunk[_chunkOffset])
|
||
Buffer.MemoryCopy(src, output + fill, take * 2, take * 2);
|
||
}
|
||
_chunkOffset += take;
|
||
fill += take;
|
||
if (_chunkOffset >= _currentChunk.Length) _currentChunk = null;
|
||
}
|
||
|
||
// Zero-fill the unfilled tail (underrun mid-response, or post-completion drain).
|
||
for (int i = fill; i < frameCount; i++) output[i] = 0;
|
||
|
||
// Completion check: drained AND writer completed AND no working chunk left.
|
||
if (_currentChunk == null
|
||
&& downstreamChannel.Reader.Count == 0
|
||
&& downstreamChannel.Reader.Completion.IsCompleted)
|
||
{
|
||
doneFlag.Set();
|
||
return StreamCallbackResult.Complete;
|
||
}
|
||
return StreamCallbackResult.Continue;
|
||
```
|
||
|
||
**Underrun behavior: zero-fill, return Continue.** If `response.done` hasn't arrived yet and the next `response.audio.delta` is just in flight, we play silence for one callback (~43 ms) instead of starving — the stream stays armed, no click on resume.
|
||
|
||
**This is the streaming-length cleanup discipline** that replaces Test 3's "scale `done.Wait` timeout to the buffer length" trick. Test 3 knew the playback wall-clock up front (`recordBuf.Length / SampleRate`); Test 4a does not, because the assistant audio length is unknown a priori. So the signaling moves from "wait this many seconds" to "wait until (writer-completed AND channel-empty AND working-chunk-empty)." The output callback is the single oracle for this triple condition.
|
||
|
||
**Startup beep wrinkle:** the beep is queued onto `downstreamChannel` BEFORE the receive loop starts producing. The output callback drains it just like any other audio. The completion check is safe because `Writer.Complete()` is only ever called on `response.done` — the beep getting drained does NOT cause an early completion firing.
|
||
|
||
## Components and files
|
||
|
||
```
|
||
tests/04a-realtime-oneshot-cs/
|
||
├── Probe.csproj
|
||
│ <TargetFramework>net9.0</TargetFramework>
|
||
│ <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
|
||
│ <SelfContained>true</SelfContained>
|
||
│ <PublishSingleFile>false</PublishSingleFile>
|
||
│ <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
|
||
│ <InvariantGlobalization>true</InvariantGlobalization>
|
||
│ <Nullable>enable</Nullable>
|
||
│ <ImplicitUsings>enable</ImplicitUsings>
|
||
│ RootNamespace=RealtimeOneShotProbe, AssemblyName=Probe
|
||
│ PackageReference PortAudioSharp2 Version="1.0.6"
|
||
│ (no Microsoft.ML.OnnxRuntime — no wakeword in 4a)
|
||
│ System.Net.WebSockets + System.Text.Json come from the BCL; no package.
|
||
│
|
||
├── Program.cs ~300 lines, top-level statements:
|
||
│ 1. Libc.setenv("PA_ALSA_PLUGHW", "1", 1) + Environment mirror
|
||
│ (no ORT env vars — no onnxruntime in 4a)
|
||
│ 2. PortAudio.Initialize
|
||
│ 3. apiKey = LoadApiKey()
|
||
│ 4. var cts = new CancellationTokenSource();
|
||
│ Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
|
||
│ 5. var ws = new ClientWebSocket();
|
||
│ ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
|
||
│ ws.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");
|
||
│ await ws.ConnectAsync(uri, cts.Token);
|
||
│ 6. Channel<short[]> upstreamChannel, downstreamChannel (both bounded 64)
|
||
│ ManualResetEventSlim doneFlag = new(false);
|
||
│ 7. Task receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel,
|
||
│ cts, _stopSending, _noMoreDeltas, transcriptBuilder));
|
||
│ 8. Send session.update (JsonSerializer.SerializeToUtf8Bytes over a dict)
|
||
│ 9. int device = FindUsbDevice();
|
||
│ 10. Open + start input stream (24 kHz, 1920 frames/cb)
|
||
│ 11. Open + start output stream (24 kHz, 1024 frames/cb)
|
||
│ 12. Task sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel,
|
||
│ cts, _stopSending));
|
||
│ 13. Queue MakeBeep(880, 0.2, 24000) onto downstreamChannel; sleep 300 ms;
|
||
│ _micArmed = true; Console.WriteLine("Speak now.");
|
||
│ 14. try { doneFlag.Wait(cts.Token); }
|
||
│ catch (OperationCanceledException) { /* Ctrl-C path */ }
|
||
│ 15. finally:
|
||
│ ws.CloseAsync(NormalClosure, "done", linkedCtsWith2sTimeout.Token)
|
||
│ inputStream.Stop; outputStream.Stop;
|
||
│ inputStream.Dispose; outputStream.Dispose;
|
||
│ PortAudio.Terminate;
|
||
│ Helpers (all in Program.cs):
|
||
│ - LoadApiKey() — reads ~/.openai_key, perms-warns, sanity
|
||
│ - MicCallback — discards unless _micArmed && !_stopSending
|
||
│ - OutputCallback — drain-or-zero-fill loop above
|
||
│ - MakeBeep(hz, sec, rate) — sin wave Int16 (lifted from Test 3, retuned 24 kHz)
|
||
│ - UpstreamSendLoop(...)
|
||
│ - ReceiveLoop(...) — JsonDocument switch on "type"
|
||
│ - FindUsbDevice() — lifted from Test 3 verbatim
|
||
│ - static class Libc { setenv }
|
||
│
|
||
└── (no models/ dir, no WakewordModel.cs)
|
||
|
||
bin/probe-cs-4a Copy of bin/probe-cs-3 with substitutions:
|
||
tests/03-full-cycle-cs → tests/04a-realtime-oneshot-cs
|
||
~/probe-cs-3 → ~/probe-cs-4a
|
||
PUBLISH_DIR /tmp/probe-cs-3-out → /tmp/probe-cs-4a-out
|
||
ssh -t retained (Ctrl-C propagation).
|
||
Wipe-and-replace publish dir retained (partial-deploy
|
||
footgun from Test 3).
|
||
Adds ONE new precondition step before publishing:
|
||
sshpass -p "$PI_PASS" ssh "$PI_USER@$PI_HOST" \
|
||
'test -f ~/.openai_key && test -r ~/.openai_key' \
|
||
|| { echo "Create ~/.openai_key on the Pi (chmod 600),
|
||
paste your sk-... key into it, then re-run."; \
|
||
exit 1; }
|
||
This catches the silent "key missing" failure mode at
|
||
the deploy step rather than 30 s into the publish + scp.
|
||
```
|
||
|
||
## Concurrency model
|
||
|
||
Five concurrent threads of execution in steady state:
|
||
|
||
1. **Main thread.** Setup, session.update send, beep + arm, `doneFlag.Wait`, teardown via `finally`.
|
||
2. **ReceiveLoop** (`Task.Run`). Pulls WS messages, dispatches on `type`. Single writer of `downstreamChannel`, `_stopSending`, `_noMoreDeltas`, transcript builder. Calls `cts.Cancel()` on error.
|
||
3. **UpstreamSendLoop** (`Task.Run`). Pulls mic frames from `upstreamChannel`, JSON-serializes `input_audio_buffer.append`, calls `ws.SendAsync`. Single writer of the WS send stream.
|
||
4. **PortAudio input audio thread** (managed by the library). Runs the mic callback every 40 ms. Allocates `short[1920]`, memcpy's the input, `upstreamChannel.Writer.TryWrite(frame)`. Never blocks.
|
||
5. **PortAudio output audio thread** (managed by the library). Runs the output callback every 43 ms. Drains `downstreamChannel` into the output buffer; zero-fills underruns; sets `doneFlag` on completion. Never blocks.
|
||
|
||
Note that `ws.SendAsync` and `ws.ReceiveAsync` can both be active concurrently — `ClientWebSocket` supports one concurrent send + one concurrent receive (but NOT multiple concurrent sends or multiple concurrent receives). Our design uses exactly one of each, so this is safe.
|
||
|
||
## Error handling
|
||
|
||
| Failure mode | Behavior | Exit |
|
||
|-|-|-|
|
||
| `~/.openai_key` missing on Pi | `bin/probe-cs-4a` precondition check exits before publishing. | 1 |
|
||
| `~/.openai_key` exists but empty / not `sk-`-shaped | `LoadApiKey()` logs `[error] API key file empty` / `[error] API key doesn't start with sk-`, exits. | 2 |
|
||
| `ws.ConnectAsync` throws (DNS, TLS, 401, 403, 429, 500, network down) | Log `[error] WS connect failed: {ex.Message}` plus inner `HttpRequestException.StatusCode` if present. | 3 |
|
||
| Server `error` event mid-session | ReceiveLoop logs the full event JSON, `cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`; `finally` runs CloseAsync (no-throw with timeout) + stream.Stop + PortAudio.Terminate. | 4 |
|
||
| WS drops mid-conversation (`WebSocketException` in ReceiveLoop) | Same as above — log + cancel + cleanup-via-finally. | 4 |
|
||
| PortAudio input stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 5 |
|
||
| PortAudio output stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 6 |
|
||
| Server takes forever and never sends `response.done` | No timeout in 4a. User's recourse is Ctrl-C. (If verification surfaces this, add a configurable timeout in 4b — out of scope for 4a.) | 0 (via Ctrl-C) |
|
||
| Mic callback can't `TryWrite` to `upstreamChannel` (UpstreamSendLoop falling behind) | Log `[status] upstream behind, dropping mic frame`, continue. (Same shape as Test 3's "consumer behind" log.) | 0 |
|
||
| Workstation Ctrl-C | `Console.CancelKeyPress` → `e.Cancel = true; cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`. `finally` runs full cleanup path. | 0 |
|
||
|
||
The `finally` block at the bottom of `Program.cs`'s outer `try` runs **once** and is the single point where WS is closed, streams are stopped, and PortAudio is terminated. Cleanup is idempotent (`stream.Stop` is a no-op if already stopped; `PortAudio.Terminate` is fine to call once even if some path didn't run).
|
||
|
||
## Gotchas carried forward (from findings.md)
|
||
|
||
1. **`Libc.setenv("PA_ALSA_PLUGHW", "1", 1)`** before any PortAudio init. `Environment.SetEnvironmentVariable` alone does NOT propagate to libc on Linux. Apply alongside the managed call.
|
||
2. **No OrtEnv work in 4a** — no `Microsoft.ML.OnnxRuntime` dependency. (4b reintroduces it.)
|
||
3. **`using Stream = PortAudioSharp.Stream;`** at the top of `Program.cs` to disambiguate from `System.IO.Stream` that `ImplicitUsings` pulls in.
|
||
4. **`<AllowUnsafeBlocks>true</AllowUnsafeBlocks>`** in `Probe.csproj` for the input/output callbacks' `Buffer.MemoryCopy` over `short*`.
|
||
5. **Fire-and-forget output stream lifetime.** `stream.Start()` runs synchronously under try/catch BEFORE any auxiliary cleanup task launches; on `Start()` failure, dispose + rethrow without ever leaking the stream. Test 4a uses a *persistent* output stream (opened once, drained for both the beep and the response audio, stopped on shutdown), so there's no per-cycle cleanup task — the `Start()` failure path is the only one that matters.
|
||
6. **Streaming-length cleanup discipline.** Test 3's "scale `done.Wait(timeout)` to recordBuf length" trick does NOT apply because the assistant audio length is unknown a priori. Replaced by the output callback's `_currentChunk == null && Reader.Count == 0 && Reader.Completion.IsCompleted` triple check. The `ManualResetEventSlim doneFlag` carries the signal to the main thread.
|
||
7. **`ssh -t` in `bin/probe-cs-4a`** for Ctrl-C propagation from workstation SIGINT.
|
||
8. **Always wipe-and-replace the whole publish directory.** A partial `scp Probe` after a rebuild leaves the old `.dll` on the target, which the new launcher loads. `bin/probe-cs-4a` does `ssh "rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a"` BEFORE the `scp -r` (Test 3 pattern).
|
||
9. **Startup beep must not be captured by the mic and shipped to OpenAI.** The `_micArmed` flag stays `false` until ~300 ms after the beep is queued to `downstreamChannel` (200 ms beep + 100 ms drain margin). The mic callback discards frames while `!_micArmed`.
|
||
|
||
New for 4a (none carried forward from prior tests — these are surfaced here for the first time):
|
||
|
||
10. **`ClientWebSocket` concurrency rule:** one concurrent Send + one concurrent Receive is supported; multiple concurrent Sends (or Receives) is not. Our design has exactly one SendLoop and one ReceiveLoop, so this is safe by construction — but if a future change adds a second sender (e.g. a ping task), it must serialize with the existing one.
|
||
11. **Channel completion semantics:** `Writer.Complete()` makes `Reader.Completion.IsCompleted` true ONLY after all written items are consumed. The output callback's triple check accounts for this correctly. Don't change to a different completion sentinel without re-verifying the corner case.
|
||
12. **API key file ownership:** the precondition check in `bin/probe-cs-4a` verifies *presence* and *readability* by `pi`, not strict mode 600. The probe itself logs a non-fatal warning if the mode is looser than 600; this lets a slightly looser local setup proceed.
|
||
|
||
## Verification (hardware only, no automated tests)
|
||
|
||
One-time setup on the Pi (workstation tells you to do this; `bin/probe-cs-4a` precondition-checks it before publishing):
|
||
|
||
```sh
|
||
sshpass -p 'assistant' ssh pi@192.168.50.115 \
|
||
'umask 077 && printf "%s\n" "sk-PASTE-YOUR-KEY-HERE" > ~/.openai_key'
|
||
```
|
||
|
||
Run from workstation:
|
||
|
||
```sh
|
||
./bin/probe-cs-4a
|
||
```
|
||
|
||
Expected console flow (abridged):
|
||
|
||
```
|
||
>> precondition: ~/.openai_key present on Pi. OK.
|
||
>> dotnet publish (linux-arm64, self-contained)
|
||
>> scp to pi@192.168.50.115:~/probe-cs-4a/ (wipe-and-replace)
|
||
>> ssh + run on Pi (Ctrl-C from this terminal stops the probe)
|
||
[ws] connecting to wss://api.openai.com/v1/realtime?model=gpt-realtime
|
||
[ws] connected
|
||
[ws] session.created sess_xxxxxxxxxxxx
|
||
[ws] session.updated — session pinned
|
||
Using device 3 ('USB Speaker Phone (LISTENAI ...)')
|
||
[beep]
|
||
Speak now.
|
||
[vad] speech started
|
||
[vad] speech stopped
|
||
[ws] response.created
|
||
[audio] first delta received (t = … s since speech_stopped)
|
||
[audio] playback drained
|
||
[transcript] Hello! How can I help you today?
|
||
[ws] closing
|
||
bye
|
||
```
|
||
|
||
### Pass criteria
|
||
|
||
All must hold during a single run on the Pi:
|
||
|
||
1. **End-to-end round-trip works once.** Probe boots, beeps, you speak a short sentence, the assistant's spoken reply plays back recognisably through the same USB Speaker Phone, the probe exits 0.
|
||
2. **Response is intelligible.** The assistant's audio is clear speech, no clipping, no dropouts longer than a single 43 ms callback (one zero-fill is fine and expected on the first chunk arrival).
|
||
3. **Round-trip latency.** Time from `[vad] speech stopped` log to `[audio] first delta received` log (i.e. first response audio byte *arriving* at the probe — actual speaker emission adds a fixed ~43 ms output-callback period on top) is **under ~2 s**. Slower is acceptable if explicitly logged; **over ~5 s is a fail.**
|
||
4. **No `[status] upstream behind` warnings** during normal operation.
|
||
5. **No `[error] WS …` events** during normal operation.
|
||
6. **No PortAudio `InputOverflow` status flags** during normal operation.
|
||
7. **Workstation Ctrl-C** at any point produces a clean exit: WS closes with normal closure, no stack trace, terminal returns to prompt. Process exit code 0 on Ctrl-C during steady listen; non-zero on Ctrl-C during a failure path is fine.
|
||
8. **`response.done`-followed-cleanup signaling works.** Probe exits within ~500 ms of the last audio sample being heard. No long hang.
|
||
|
||
### What success teaches us for 4b
|
||
|
||
- Auth + WS handshake + headers + URL are correct.
|
||
- `session.update` with `server_vad` works; the server fires `speech_started` / `speech_stopped` correctly on real Pi mic input.
|
||
- Base64 decode of `response.audio.delta` produces playable PCM16 at 24 kHz on the USB Speaker Phone (PortAudio plug-resamples 48↔24 at ALSA without trouble).
|
||
- The streaming-length cleanup discipline (`_noMoreDeltas` + `Channel.Completion.IsCompleted` + working-chunk check) is correct.
|
||
- The `Channel<short[]>` upstream/downstream pattern handles backpressure cleanly.
|
||
- `bin/probe-cs-4a`'s `~/.openai_key` precondition pattern is the right key-management story (it'll lift verbatim into 4b and the main assistant).
|
||
|
||
### What failure would tell us
|
||
|
||
- **401 / 403 on connect:** API key is wrong or the file's been mangled.
|
||
- **WS connects but `session.updated` never arrives:** the session config is malformed for the current Realtime API surface — `modalities`, `turn_detection`, or `voice` is using an outdated field name or value.
|
||
- **No `speech_stopped` event ever fires:** `turn_detection` config is wrong for the current API (field renamed?), or the mic is silent (gain too low, wrong device, ALSA plug-rate-conversion broken at 24 kHz). Cross-check by adding a periodic RMS log in the mic callback.
|
||
- **`[status] upstream behind` repeats:** UpstreamSendLoop is slower than the mic callback. Most likely `ws.SendAsync` is blocking on TCP. Cheap remediation: bump channel capacity. Real remediation: investigate the WS backpressure.
|
||
- **First delta takes >5 s:** server-side issue OR our `instructions` is producing a long thinking pause. Tighten instructions or report and move on.
|
||
- **Garbled / chipmunk-speed playback:** sample-rate mismatch. Verify the output stream actually opened at 24 kHz (PortAudio might have silently substituted a default).
|
||
- **Click at the start of playback:** zero-fill underrun ran before the first delta. Acceptable for a probe; if it bothers you, add a small pre-roll buffer.
|
||
- **Probe hangs on exit:** `Writer.Complete()` was never called (receive loop missed `response.done`), or the output callback's triple check is wrong. Investigate via logging.
|
||
|
||
## Reference paths
|
||
|
||
- Spec (this doc): `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`
|
||
- Plan (next, to write): `docs/superpowers/plans/2026-06-12-test-4a-openai-realtime.md`
|
||
- Code (to be written): `tests/04a-realtime-oneshot-cs/`
|
||
- Deploy (to be written): `bin/probe-cs-4a`
|
||
- Test 3 reference: `tests/03-full-cycle-cs/Program.cs`, `bin/probe-cs-3`, `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md`
|
||
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
|
||
- API key file location: `/home/pi/.openai_key` on the Pi, mode 600, single line of `sk-…`
|
||
- Canonical findings: `findings.md` § "C# probe outcome" through § "C# full-cycle outcome"
|
||
- OpenAI Realtime API surface as of this brainstorm:
|
||
- WebSocket URL: `wss://api.openai.com/v1/realtime?model=gpt-realtime`
|
||
- Headers: `Authorization: Bearer <key>`, `OpenAI-Beta: realtime=v1`
|
||
- Audio formats: `pcm16` (default 24 kHz Int16), `g711_ulaw`, `g711_alaw`
|
||
- Session config keys used: `modalities`, `instructions`, `voice`, `input_audio_format`, `output_audio_format`, `turn_detection`
|
||
- `turn_detection.server_vad` subkeys used: `threshold`, `prefix_padding_ms`, `silence_duration_ms`, `create_response`
|
||
- Server events consumed: `session.created`, `session.updated`, `input_audio_buffer.speech_started`, `input_audio_buffer.speech_stopped`, `input_audio_buffer.committed`, `response.created`, `response.audio_transcript.delta`, `response.audio.delta`, `response.audio.done`, `response.done`, `error`
|