From dea126f994fa55275e326e489b4c9c0f653fca1e Mon Sep 17 00:00:00 2001 From: Assistant builder Date: Fri, 12 Jun 2026 16:59:10 +0000 Subject: [PATCH] =?UTF-8?q?Test=204a:=20spec=20=E2=80=94=20OpenAI=20Realti?= =?UTF-8?q?me=20API=20one-shot=20voice=20round-trip?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Boot → beep → user speaks → server VAD ends turn → assistant audio plays back → exit 0. No wakeword, no loop, no state machine. Splits Test 4 into 4a (this spec) and 4b (wakeword + loop + 16↔24 kHz coexistence, future). 4a debugs the OpenAI Realtime surface — auth, WS handshake, session.update payload, server VAD, base64 audio, streaming- length output-stream cleanup — in isolation from 4b's harder concurrency. --- ...26-06-12-test-4a-openai-realtime-design.md | 474 ++++++++++++++++++ 1 file changed, 474 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md diff --git a/docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md b/docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md new file mode 100644 index 0000000..babed58 --- /dev/null +++ b/docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md @@ -0,0 +1,474 @@ +# Test 4a — OpenAI Realtime API one-shot voice round-trip (C#) + +A throwaway C# probe that has a single round-trip spoken conversation with the OpenAI Realtime API: + +> Boot → beep → user speaks one sentence → server VAD ends the turn → assistant audio reply plays back through the same USB Speaker Phone → exit 0. + +No wakeword. No loop. No state machine. One process, runs once, dies. + +Test 4a's job is to debug the OpenAI Realtime API surface (auth, WS handshake, `session.update` payload, event JSON shapes, base64 audio decode, server VAD endpointing, streaming-length output-stream cleanup) **in isolation** from the harder concurrency story of Test 4b (wakeword + loop + 16↔24 kHz coexistence). + +Read `findings.md` (§ "C# probe outcome", § "C# wakeword probe outcome", § "C# full-cycle outcome") before reading this. Everything below assumes those outcomes as ground truth. The patterns lifted directly from Test 3 are: `Libc.setenv`, the `using Stream = PortAudioSharp.Stream;` import disambiguation, `true`, the fire-and-forget output stream lifetime (Start under try/catch BEFORE the cleanup task launches), `FindUsbDevice`, the `MakeBeep` helper, `bin/probe-cs-3`'s wipe-and-replace publish shape with `ssh -t`. + +## Goal + +A single-binary C# probe that: + +1. Reads the OpenAI API key from `~/.openai_key` on the Pi (mode 600). +2. Connects to `wss://api.openai.com/v1/realtime?model=gpt-realtime` over a WebSocket with `Authorization: Bearer …` + `OpenAI-Beta: realtime=v1` headers. +3. Sends one `session.update` event pinning audio formats, voice, instructions, and `turn_detection: server_vad`. +4. Plays a startup beep through the USB Speaker Phone. +5. Opens a 24 kHz mono Int16 input stream on the same device; streams mic audio to OpenAI as `input_audio_buffer.append` events. +6. Waits for the server VAD to fire `input_audio_buffer.speech_stopped` (server then auto-commits the buffer and creates the response). +7. Stops sending mic audio at that point. Receives `response.audio.delta` chunks, base64-decodes them, plays them through a 24 kHz output stream on the same device. +8. On `response.done` AND the output buffer drained, closes the WebSocket cleanly, terminates PortAudio, exits 0. +9. Exits cleanly on Ctrl-C from the workstation (SIGINT propagated via `ssh -t`) at any stage. + +## Non-goals + +- **No wakeword.** No `WakewordModel.cs`. No 16 kHz audio path anywhere in 4a. (Test 4b layers that on top of this proven base.) +- **No state machine, no loop.** Probe runs once and exits. +- **No barge-in.** Once `speech_stopped` fires, mic frames are dropped on the floor. The user cannot interrupt the assistant in 4a. +- **No client-side resampling.** Mic = 24 kHz, output = 24 kHz, OpenAI = 24 kHz pcm16. PortAudio plug-resamples 48↔24 at ALSA on the USB device. +- **No WAV capture.** Logging is stdout/stderr only. +- **No retry on WS errors, no reconnect, no health check.** Fail loud, exit non-zero, log enough to diagnose. +- **No automated tests.** Hardware verification only. + +## Architecture + +``` + ┌──────────────────────────────────────┐ + │ ClientWebSocket │ + │ wss://api.openai.com/v1/realtime? │ + │ model=gpt-realtime │ + │ Headers: Authorization: Bearer … │ + │ OpenAI-Beta: realtime=v1 │ + └────┬──────────────────────────┬──────┘ + │ SendAsync │ ReceiveAsync + ▼ ▼ + ┌──────────────────────────┐ ┌──────────────────────────────┐ + │ UpstreamSendLoop │ │ ReceiveLoop (Task.Run) │ + │ (Task.Run) │ │ while (!cts.Cancel) { │ + │ drain upstreamChannel: │ │ msg = ReceiveAsync(…) │ + │ pull short[1920] │ │ doc = JsonDocument.Parse │ + │ → base64 │ │ switch (doc["type"]) │ + │ → input_audio_ │ │ session.created → log │ + │ buffer.append │ │ session.updated → log │ + │ → ws.SendAsync │ │ speech_started → log │ + │ exits when │ │ speech_stopped → │ + │ _stopSending && empty│ │ _stopSending = true │ + └──────────┬───────────────┘ │ response.audio.delta: │ + │ (FullMode.Wait) │ base64 → short[] │ + │ │ → downstreamChannel │ + ▲ │ .Writer.WriteAsync │ + │ short[1920] │ audio_transcript.delta:│ + │ (40 ms @ 24 kHz) │ append to log buf │ + ┌──────────┴───────────────┐ │ response.done → │ + │ PortAudio input stream │ │ _noMoreDeltas = true │ + │ 24 kHz mono Int16, │ │ downstreamChannel │ + │ 1920 frames/cb │ │ .Writer.Complete │ + │ callback: │ │ error → log + Cancel │ + │ if !_micArmed: discard│ └────────────┬─────────────────┘ + │ elif _stopSending: │ │ + │ discard │ ▼ + │ else: │ ┌──────────────────────────────┐ + │ copy → short[] │ │ Channel │ + │ TryWrite to │ │ (bounded cap 64, │ + │ upstreamChannel │ │ FullMode.Wait) │ + └──────────────────────────┘ └────────────┬─────────────────┘ + │ + ▼ + ┌──────────────────────────────────┐ + │ PortAudio output stream │ + │ 24 kHz mono Int16, │ + │ 1024 frames/cb │ + │ callback: │ + │ fill = 0 │ + │ while fill < frameCount: │ + │ if currentChunk == null: │ + │ try pull short[] or break │ + │ take = min(remaining, slack)│ + │ memcpy chunk → output │ + │ zero-fill remaining │ + │ if currentChunk == null │ + │ && Reader.Count == 0 │ + │ && Completion.IsCompleted: │ + │ doneFlag.Set() │ + │ return Complete │ + │ return Continue │ + └────────────┬─────────────────────┘ + │ + ▼ + main thread: doneFlag.Wait + → ws.CloseAsync(NormalClosure) + → input/output stream.Stop + → PortAudio.Terminate + → exit 0 +``` + +One process. Three managed threads (main + UpstreamSendLoop + ReceiveLoop) plus PortAudio's input and output audio threads (managed by the library). + +**Shared mutable state — the full list, small on purpose:** + +- `Channel upstreamChannel`, bounded 64, `FullMode.Wait`, single-writer (mic callback), single-reader (UpstreamSendLoop). Holds mic frames waiting to be base64'd and sent. +- `Channel downstreamChannel`, bounded 64, `FullMode.Wait`, **multi-writer** (main thread writes the startup beep; ReceiveLoop writes response audio chunks; temporally separated but Channel's `SingleWriter` flag is a contract not a runtime check, so keep it `false`), single-reader (output callback). Holds decoded audio chunks waiting to be played. Its `Writer.Complete()` (called by ReceiveLoop on `response.done`) is the WS-stream-end signal for the output callback. +- `volatile bool _micArmed`: starts `false`; set `true` after the startup beep has been queued and drained. Mic frames before this point are discarded so the beep doesn't get captured and shipped to OpenAI. +- `volatile bool _stopSending`: set by ReceiveLoop on `input_audio_buffer.speech_stopped`. Read by both the mic callback (which then discards) and UpstreamSendLoop (which then drains its queue and exits). +- `volatile bool _noMoreDeltas`: set by ReceiveLoop on `response.done`. Not strictly required because `downstreamChannel.Writer.Complete()` carries the same signal — but kept as a redundant fast path that the output callback can check without touching the channel. +- `ManualResetEventSlim doneFlag`: set by the output callback when it has drained the channel after completion. Awaited by main. +- `CancellationTokenSource cts`: wired to `Console.CancelKeyPress` + any WS error. Cancels both Task.Run loops; main's `doneFlag.Wait(cts.Token)` honours it. + +No locks. No `Interlocked`. The bounded `Channel` handles all the cross-thread audio handoff with the right backpressure semantics. + +## WebSocket lifecycle + +Six discrete phases, all in `Program.cs` top-level statements + the two `Task.Run` loops: + +1. **Load API key.** `LoadApiKey()` helper reads `~/.openai_key`, trims whitespace. Mode 600 verified — non-fatal warning if looser. Fatal error (exit 2) if the file is missing, empty, or doesn't start with `sk-`. +2. **Construct `ClientWebSocket`** with two request headers: `Authorization: Bearer ` and `OpenAI-Beta: realtime=v1`. `ConnectAsync` to `wss://api.openai.com/v1/realtime?model=gpt-realtime` with `cts.Token`. On failure (DNS, TLS, 401, 403, 429, 5xx), log the exception message + any HTTP status carried on `WebSocketException.WebSocketErrorCode` / inner `HttpRequestException.StatusCode`; exit 3. +3. **Launch ReceiveLoop** (`Task.Run`) before sending anything, so we don't miss `session.created`. +4. **Send `session.update`.** First client message; pins the session config (see "Client → server events" below). Don't strictly block on `session.updated` — the receive loop logs it asynchronously — but the spec expects it within ~200 ms. +5. **Open + start PortAudio streams.** Input stream first (24 kHz mono Int16, 1920 frames/cb), then output stream (24 kHz mono Int16, 1024 frames/cb). Both opened on `FindUsbDevice()`. Launch UpstreamSendLoop (`Task.Run`). +6. **Startup beep + arm mic.** Queue a 200 ms 880 Hz beep (`MakeBeep` lifted from Test 3, regenerated at 24 kHz) onto `downstreamChannel`. Sleep ~300 ms (beep + drain margin). Set `_micArmed = true`. Print `Speak now.` to stdout. +7. **Wait for `doneFlag`** on the main thread (with `cts.Token` for Ctrl-C). On wake-up: `ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", …)` with a 2 s timeout token. Stop input stream, stop output stream, terminate PortAudio, exit 0. + +## Client → server event payloads + +Three send-once messages on the main thread, plus a stream of `input_audio_buffer.append` from UpstreamSendLoop. All written with `JsonSerializer.Serialize` over `Dictionary` (lightest thing that doesn't require typed records for a probe). + +```json +// (1) Pinned session config — first thing sent after WS connect. +{ + "type": "session.update", + "session": { + "modalities": ["audio", "text"], + "instructions": "Reply in one short sentence.", + "voice": "alloy", + "input_audio_format": "pcm16", + "output_audio_format": "pcm16", + "turn_detection": { + "type": "server_vad", + "threshold": 0.5, + "prefix_padding_ms": 300, + "silence_duration_ms": 500, + "create_response": true + } + } +} + +// (2) Per mic frame (~40 ms cadence), from UpstreamSendLoop: +{ "type": "input_audio_buffer.append", "audio": "" } +``` + +No `conversation.item.create` and no `response.create` from the client — server VAD's `create_response: true` makes the server commit and create the response automatically on `speech_stopped`. + +`instructions: "Reply in one short sentence."` keeps the response audio bounded (~2-4 s of speech) so a probe run is fast to verify by ear and cheap on quota. + +`voice: "alloy"` is the long-standing default voice and is documented as universally available for raw WS sessions. + +## Server → client event handlers + +ReceiveLoop uses a 4 KB receive buffer + standard `ReceiveAsync` loop with `EndOfMessage` accumulation into a `MemoryStream`. On message end, `JsonDocument.Parse`. Switch on the top-level `type` property's string value: + +| `type` | Action | +|-|-| +| `session.created` | log `[ws] session.created ` | +| `session.updated` | log `[ws] session.updated — session pinned` | +| `input_audio_buffer.speech_started` | log `[vad] speech started` | +| `input_audio_buffer.speech_stopped` | log `[vad] speech stopped`; set `_stopSending = true` | +| `input_audio_buffer.committed` | log only — server confirms it sealed the buffer | +| `response.created` | log only | +| `response.audio_transcript.delta` | append `delta` to an accumulating `StringBuilder` | +| `response.audio.delta` | base64-decode `delta` → bytes → reinterpret as `short[]` → `await downstreamChannel.Writer.WriteAsync(samples, cts.Token)`. The `await` is the backpressure path: if the channel is full, this blocks the receive loop, which blocks the WS receive buffer — correct behavior for a slow speaker. | +| `response.audio.done` | log only | +| `response.done` | set `_noMoreDeltas = true`; `downstreamChannel.Writer.Complete()`; print the accumulated transcript | +| `error` | log the full event JSON; `cts.Cancel()` | +| anything else | log `[ws] ignored type=<…>` once (or at debug verbosity), continue | + +## Audio buffering — input + +- Mic callback runs every 40 ms (1920 samples @ 24 kHz mono Int16, matches `framesPerBuffer = 1920`). +- Callback allocates `short[1920]`, `Buffer.MemoryCopy`s the input pointer, `upstreamChannel.Writer.TryWrite(frame)`. +- On TryWrite failure (channel full = UpstreamSendLoop is behind): log `[status] upstream behind, dropping mic frame`, continue. +- Discard frames when `_micArmed == false` (during startup beep) or `_stopSending == true` (after `speech_stopped`). +- On the `InputOverflow` status flag from PortAudio: log `[status] input overflow`, continue. + +UpstreamSendLoop pseudocode: + +``` +await ReceiveLoop signals session.updated (or just optimistically loop from start) +while (!cts.IsCancellationRequested): + if _stopSending && upstreamChannel.Reader.Count == 0: break + frame = await upstreamChannel.Reader.ReadAsync(cts.Token) + if _stopSending: continue // drop frames captured after server VAD cutoff + bytes = MemoryMarshal.AsBytes(frame.AsSpan()) + base64 = Convert.ToBase64String(bytes) + json = JsonSerializer.SerializeToUtf8Bytes(new { + type = "input_audio_buffer.append", + audio = base64 + }) + await ws.SendAsync(json, WebSocketMessageType.Text, endOfMessage: true, cts.Token) +``` + +## Audio buffering — output + +`downstreamChannel = Channel.CreateBounded(new BoundedChannelOptions(64) { FullMode = BoundedChannelFullMode.Wait, SingleWriter = false, SingleReader = true })`. `SingleWriter = false` because the main thread queues the startup beep AND ReceiveLoop queues response audio chunks (temporally separated, but the flag is a hard contract). + +Capacity rationale: 64 chunks × ~40-80 ms per `response.audio.delta` ≈ 2.5-5 s of buffered audio. Comfortable headroom against network jitter without holding silly amounts of memory. + +ReceiveLoop writes `short[]` chunks of varying length (one per `response.audio.delta` event). On `response.done`, calls `downstreamChannel.Writer.Complete()`. + +Output callback's job is to fill `frameCount` shorts every callback (1024 frames = ~43 ms @ 24 kHz). It maintains a `short[]? _currentChunk` + `int _chunkOffset` across calls. Pseudocode: + +``` +int fill = 0; +while (fill < frameCount) { + if (_currentChunk == null) { + if (!downstreamChannel.Reader.TryRead(out _currentChunk)) break; // drained + _chunkOffset = 0; + } + int slack = frameCount - fill; + int remaining = _currentChunk.Length - _chunkOffset; + int take = Math.Min(remaining, slack); + unsafe { + fixed (short* src = &_currentChunk[_chunkOffset]) + Buffer.MemoryCopy(src, output + fill, take * 2, take * 2); + } + _chunkOffset += take; + fill += take; + if (_chunkOffset >= _currentChunk.Length) _currentChunk = null; +} + +// Zero-fill the unfilled tail (underrun mid-response, or post-completion drain). +for (int i = fill; i < frameCount; i++) output[i] = 0; + +// Completion check: drained AND writer completed AND no working chunk left. +if (_currentChunk == null + && downstreamChannel.Reader.Count == 0 + && downstreamChannel.Reader.Completion.IsCompleted) +{ + doneFlag.Set(); + return StreamCallbackResult.Complete; +} +return StreamCallbackResult.Continue; +``` + +**Underrun behavior: zero-fill, return Continue.** If `response.done` hasn't arrived yet and the next `response.audio.delta` is just in flight, we play silence for one callback (~43 ms) instead of starving — the stream stays armed, no click on resume. + +**This is the streaming-length cleanup discipline** that replaces Test 3's "scale `done.Wait` timeout to the buffer length" trick. Test 3 knew the playback wall-clock up front (`recordBuf.Length / SampleRate`); Test 4a does not, because the assistant audio length is unknown a priori. So the signaling moves from "wait this many seconds" to "wait until (writer-completed AND channel-empty AND working-chunk-empty)." The output callback is the single oracle for this triple condition. + +**Startup beep wrinkle:** the beep is queued onto `downstreamChannel` BEFORE the receive loop starts producing. The output callback drains it just like any other audio. The completion check is safe because `Writer.Complete()` is only ever called on `response.done` — the beep getting drained does NOT cause an early completion firing. + +## Components and files + +``` +tests/04a-realtime-oneshot-cs/ +├── Probe.csproj +│ net9.0 +│ linux-arm64 +│ true +│ false +│ true +│ true +│ enable +│ enable +│ RootNamespace=RealtimeOneShotProbe, AssemblyName=Probe +│ PackageReference PortAudioSharp2 Version="1.0.6" +│ (no Microsoft.ML.OnnxRuntime — no wakeword in 4a) +│ System.Net.WebSockets + System.Text.Json come from the BCL; no package. +│ +├── Program.cs ~300 lines, top-level statements: +│ 1. Libc.setenv("PA_ALSA_PLUGHW", "1", 1) + Environment mirror +│ (no ORT env vars — no onnxruntime in 4a) +│ 2. PortAudio.Initialize +│ 3. apiKey = LoadApiKey() +│ 4. var cts = new CancellationTokenSource(); +│ Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); }; +│ 5. var ws = new ClientWebSocket(); +│ ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}"); +│ ws.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1"); +│ await ws.ConnectAsync(uri, cts.Token); +│ 6. Channel upstreamChannel, downstreamChannel (both bounded 64) +│ ManualResetEventSlim doneFlag = new(false); +│ 7. Task receiveTask = Task.Run(() => ReceiveLoop(ws, downstreamChannel, +│ cts, _stopSending, _noMoreDeltas, transcriptBuilder)); +│ 8. Send session.update (JsonSerializer.SerializeToUtf8Bytes over a dict) +│ 9. int device = FindUsbDevice(); +│ 10. Open + start input stream (24 kHz, 1920 frames/cb) +│ 11. Open + start output stream (24 kHz, 1024 frames/cb) +│ 12. Task sendTask = Task.Run(() => UpstreamSendLoop(ws, upstreamChannel, +│ cts, _stopSending)); +│ 13. Queue MakeBeep(880, 0.2, 24000) onto downstreamChannel; sleep 300 ms; +│ _micArmed = true; Console.WriteLine("Speak now."); +│ 14. try { doneFlag.Wait(cts.Token); } +│ catch (OperationCanceledException) { /* Ctrl-C path */ } +│ 15. finally: +│ ws.CloseAsync(NormalClosure, "done", linkedCtsWith2sTimeout.Token) +│ inputStream.Stop; outputStream.Stop; +│ inputStream.Dispose; outputStream.Dispose; +│ PortAudio.Terminate; +│ Helpers (all in Program.cs): +│ - LoadApiKey() — reads ~/.openai_key, perms-warns, sanity +│ - MicCallback — discards unless _micArmed && !_stopSending +│ - OutputCallback — drain-or-zero-fill loop above +│ - MakeBeep(hz, sec, rate) — sin wave Int16 (lifted from Test 3, retuned 24 kHz) +│ - UpstreamSendLoop(...) +│ - ReceiveLoop(...) — JsonDocument switch on "type" +│ - FindUsbDevice() — lifted from Test 3 verbatim +│ - static class Libc { setenv } +│ +└── (no models/ dir, no WakewordModel.cs) + +bin/probe-cs-4a Copy of bin/probe-cs-3 with substitutions: + tests/03-full-cycle-cs → tests/04a-realtime-oneshot-cs + ~/probe-cs-3 → ~/probe-cs-4a + PUBLISH_DIR /tmp/probe-cs-3-out → /tmp/probe-cs-4a-out + ssh -t retained (Ctrl-C propagation). + Wipe-and-replace publish dir retained (partial-deploy + footgun from Test 3). + Adds ONE new precondition step before publishing: + sshpass -p "$PI_PASS" ssh "$PI_USER@$PI_HOST" \ + 'test -f ~/.openai_key && test -r ~/.openai_key' \ + || { echo "Create ~/.openai_key on the Pi (chmod 600), + paste your sk-... key into it, then re-run."; \ + exit 1; } + This catches the silent "key missing" failure mode at + the deploy step rather than 30 s into the publish + scp. +``` + +## Concurrency model + +Five concurrent threads of execution in steady state: + +1. **Main thread.** Setup, session.update send, beep + arm, `doneFlag.Wait`, teardown via `finally`. +2. **ReceiveLoop** (`Task.Run`). Pulls WS messages, dispatches on `type`. Single writer of `downstreamChannel`, `_stopSending`, `_noMoreDeltas`, transcript builder. Calls `cts.Cancel()` on error. +3. **UpstreamSendLoop** (`Task.Run`). Pulls mic frames from `upstreamChannel`, JSON-serializes `input_audio_buffer.append`, calls `ws.SendAsync`. Single writer of the WS send stream. +4. **PortAudio input audio thread** (managed by the library). Runs the mic callback every 40 ms. Allocates `short[1920]`, memcpy's the input, `upstreamChannel.Writer.TryWrite(frame)`. Never blocks. +5. **PortAudio output audio thread** (managed by the library). Runs the output callback every 43 ms. Drains `downstreamChannel` into the output buffer; zero-fills underruns; sets `doneFlag` on completion. Never blocks. + +Note that `ws.SendAsync` and `ws.ReceiveAsync` can both be active concurrently — `ClientWebSocket` supports one concurrent send + one concurrent receive (but NOT multiple concurrent sends or multiple concurrent receives). Our design uses exactly one of each, so this is safe. + +## Error handling + +| Failure mode | Behavior | Exit | +|-|-|-| +| `~/.openai_key` missing on Pi | `bin/probe-cs-4a` precondition check exits before publishing. | 1 | +| `~/.openai_key` exists but empty / not `sk-`-shaped | `LoadApiKey()` logs `[error] API key file empty` / `[error] API key doesn't start with sk-`, exits. | 2 | +| `ws.ConnectAsync` throws (DNS, TLS, 401, 403, 429, 500, network down) | Log `[error] WS connect failed: {ex.Message}` plus inner `HttpRequestException.StatusCode` if present. | 3 | +| Server `error` event mid-session | ReceiveLoop logs the full event JSON, `cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`; `finally` runs CloseAsync (no-throw with timeout) + stream.Stop + PortAudio.Terminate. | 4 | +| WS drops mid-conversation (`WebSocketException` in ReceiveLoop) | Same as above — log + cancel + cleanup-via-finally. | 4 | +| PortAudio input stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 5 | +| PortAudio output stream `Start()` throws | Log + Dispose stream + cts.Cancel. | 6 | +| Server takes forever and never sends `response.done` | No timeout in 4a. User's recourse is Ctrl-C. (If verification surfaces this, add a configurable timeout in 4b — out of scope for 4a.) | 0 (via Ctrl-C) | +| Mic callback can't `TryWrite` to `upstreamChannel` (UpstreamSendLoop falling behind) | Log `[status] upstream behind, dropping mic frame`, continue. (Same shape as Test 3's "consumer behind" log.) | 0 | +| Workstation Ctrl-C | `Console.CancelKeyPress` → `e.Cancel = true; cts.Cancel()`. Main wakes from `doneFlag.Wait(cts.Token)` with `OperationCanceledException`. `finally` runs full cleanup path. | 0 | + +The `finally` block at the bottom of `Program.cs`'s outer `try` runs **once** and is the single point where WS is closed, streams are stopped, and PortAudio is terminated. Cleanup is idempotent (`stream.Stop` is a no-op if already stopped; `PortAudio.Terminate` is fine to call once even if some path didn't run). + +## Gotchas carried forward (from findings.md) + +1. **`Libc.setenv("PA_ALSA_PLUGHW", "1", 1)`** before any PortAudio init. `Environment.SetEnvironmentVariable` alone does NOT propagate to libc on Linux. Apply alongside the managed call. +2. **No OrtEnv work in 4a** — no `Microsoft.ML.OnnxRuntime` dependency. (4b reintroduces it.) +3. **`using Stream = PortAudioSharp.Stream;`** at the top of `Program.cs` to disambiguate from `System.IO.Stream` that `ImplicitUsings` pulls in. +4. **`true`** in `Probe.csproj` for the input/output callbacks' `Buffer.MemoryCopy` over `short*`. +5. **Fire-and-forget output stream lifetime.** `stream.Start()` runs synchronously under try/catch BEFORE any auxiliary cleanup task launches; on `Start()` failure, dispose + rethrow without ever leaking the stream. Test 4a uses a *persistent* output stream (opened once, drained for both the beep and the response audio, stopped on shutdown), so there's no per-cycle cleanup task — the `Start()` failure path is the only one that matters. +6. **Streaming-length cleanup discipline.** Test 3's "scale `done.Wait(timeout)` to recordBuf length" trick does NOT apply because the assistant audio length is unknown a priori. Replaced by the output callback's `_currentChunk == null && Reader.Count == 0 && Reader.Completion.IsCompleted` triple check. The `ManualResetEventSlim doneFlag` carries the signal to the main thread. +7. **`ssh -t` in `bin/probe-cs-4a`** for Ctrl-C propagation from workstation SIGINT. +8. **Always wipe-and-replace the whole publish directory.** A partial `scp Probe` after a rebuild leaves the old `.dll` on the target, which the new launcher loads. `bin/probe-cs-4a` does `ssh "rm -rf ~/probe-cs-4a && mkdir ~/probe-cs-4a"` BEFORE the `scp -r` (Test 3 pattern). +9. **Startup beep must not be captured by the mic and shipped to OpenAI.** The `_micArmed` flag stays `false` until ~300 ms after the beep is queued to `downstreamChannel` (200 ms beep + 100 ms drain margin). The mic callback discards frames while `!_micArmed`. + +New for 4a (none carried forward from prior tests — these are surfaced here for the first time): + +10. **`ClientWebSocket` concurrency rule:** one concurrent Send + one concurrent Receive is supported; multiple concurrent Sends (or Receives) is not. Our design has exactly one SendLoop and one ReceiveLoop, so this is safe by construction — but if a future change adds a second sender (e.g. a ping task), it must serialize with the existing one. +11. **Channel completion semantics:** `Writer.Complete()` makes `Reader.Completion.IsCompleted` true ONLY after all written items are consumed. The output callback's triple check accounts for this correctly. Don't change to a different completion sentinel without re-verifying the corner case. +12. **API key file ownership:** the precondition check in `bin/probe-cs-4a` verifies *presence* and *readability* by `pi`, not strict mode 600. The probe itself logs a non-fatal warning if the mode is looser than 600; this lets a slightly looser local setup proceed. + +## Verification (hardware only, no automated tests) + +One-time setup on the Pi (workstation tells you to do this; `bin/probe-cs-4a` precondition-checks it before publishing): + +```sh +sshpass -p 'assistant' ssh pi@192.168.50.115 \ + 'umask 077 && printf "%s\n" "sk-PASTE-YOUR-KEY-HERE" > ~/.openai_key' +``` + +Run from workstation: + +```sh +./bin/probe-cs-4a +``` + +Expected console flow (abridged): + +``` +>> precondition: ~/.openai_key present on Pi. OK. +>> dotnet publish (linux-arm64, self-contained) +>> scp to pi@192.168.50.115:~/probe-cs-4a/ (wipe-and-replace) +>> ssh + run on Pi (Ctrl-C from this terminal stops the probe) +[ws] connecting to wss://api.openai.com/v1/realtime?model=gpt-realtime +[ws] connected +[ws] session.created sess_xxxxxxxxxxxx +[ws] session.updated — session pinned +Using device 3 ('USB Speaker Phone (LISTENAI ...)') +[beep] +Speak now. +[vad] speech started +[vad] speech stopped +[ws] response.created +[audio] first delta received (t = … s since speech_stopped) +[audio] playback drained +[transcript] Hello! How can I help you today? +[ws] closing +bye +``` + +### Pass criteria + +All must hold during a single run on the Pi: + +1. **End-to-end round-trip works once.** Probe boots, beeps, you speak a short sentence, the assistant's spoken reply plays back recognisably through the same USB Speaker Phone, the probe exits 0. +2. **Response is intelligible.** The assistant's audio is clear speech, no clipping, no dropouts longer than a single 43 ms callback (one zero-fill is fine and expected on the first chunk arrival). +3. **Round-trip latency.** Time from `[vad] speech stopped` log to `[audio] first delta received` log (i.e. first response audio byte *arriving* at the probe — actual speaker emission adds a fixed ~43 ms output-callback period on top) is **under ~2 s**. Slower is acceptable if explicitly logged; **over ~5 s is a fail.** +4. **No `[status] upstream behind` warnings** during normal operation. +5. **No `[error] WS …` events** during normal operation. +6. **No PortAudio `InputOverflow` status flags** during normal operation. +7. **Workstation Ctrl-C** at any point produces a clean exit: WS closes with normal closure, no stack trace, terminal returns to prompt. Process exit code 0 on Ctrl-C during steady listen; non-zero on Ctrl-C during a failure path is fine. +8. **`response.done`-followed-cleanup signaling works.** Probe exits within ~500 ms of the last audio sample being heard. No long hang. + +### What success teaches us for 4b + +- Auth + WS handshake + headers + URL are correct. +- `session.update` with `server_vad` works; the server fires `speech_started` / `speech_stopped` correctly on real Pi mic input. +- Base64 decode of `response.audio.delta` produces playable PCM16 at 24 kHz on the USB Speaker Phone (PortAudio plug-resamples 48↔24 at ALSA without trouble). +- The streaming-length cleanup discipline (`_noMoreDeltas` + `Channel.Completion.IsCompleted` + working-chunk check) is correct. +- The `Channel` upstream/downstream pattern handles backpressure cleanly. +- `bin/probe-cs-4a`'s `~/.openai_key` precondition pattern is the right key-management story (it'll lift verbatim into 4b and the main assistant). + +### What failure would tell us + +- **401 / 403 on connect:** API key is wrong or the file's been mangled. +- **WS connects but `session.updated` never arrives:** the session config is malformed for the current Realtime API surface — `modalities`, `turn_detection`, or `voice` is using an outdated field name or value. +- **No `speech_stopped` event ever fires:** `turn_detection` config is wrong for the current API (field renamed?), or the mic is silent (gain too low, wrong device, ALSA plug-rate-conversion broken at 24 kHz). Cross-check by adding a periodic RMS log in the mic callback. +- **`[status] upstream behind` repeats:** UpstreamSendLoop is slower than the mic callback. Most likely `ws.SendAsync` is blocking on TCP. Cheap remediation: bump channel capacity. Real remediation: investigate the WS backpressure. +- **First delta takes >5 s:** server-side issue OR our `instructions` is producing a long thinking pause. Tighten instructions or report and move on. +- **Garbled / chipmunk-speed playback:** sample-rate mismatch. Verify the output stream actually opened at 24 kHz (PortAudio might have silently substituted a default). +- **Click at the start of playback:** zero-fill underrun ran before the first delta. Acceptable for a probe; if it bothers you, add a small pre-roll buffer. +- **Probe hangs on exit:** `Writer.Complete()` was never called (receive loop missed `response.done`), or the output callback's triple check is wrong. Investigate via logging. + +## Reference paths + +- Spec (this doc): `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md` +- Plan (next, to write): `docs/superpowers/plans/2026-06-12-test-4a-openai-realtime.md` +- Code (to be written): `tests/04a-realtime-oneshot-cs/` +- Deploy (to be written): `bin/probe-cs-4a` +- Test 3 reference: `tests/03-full-cycle-cs/Program.cs`, `bin/probe-cs-3`, `docs/superpowers/specs/2026-06-12-test-3-full-cycle-csharp-design.md` +- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`) +- API key file location: `/home/pi/.openai_key` on the Pi, mode 600, single line of `sk-…` +- Canonical findings: `findings.md` § "C# probe outcome" through § "C# full-cycle outcome" +- OpenAI Realtime API surface as of this brainstorm: + - WebSocket URL: `wss://api.openai.com/v1/realtime?model=gpt-realtime` + - Headers: `Authorization: Bearer `, `OpenAI-Beta: realtime=v1` + - Audio formats: `pcm16` (default 24 kHz Int16), `g711_ulaw`, `g711_alaw` + - Session config keys used: `modalities`, `instructions`, `voice`, `input_audio_format`, `output_audio_format`, `turn_detection` + - `turn_detection.server_vad` subkeys used: `threshold`, `prefix_padding_ms`, `silence_duration_ms`, `create_response` + - Server events consumed: `session.created`, `session.updated`, `input_audio_buffer.speech_started`, `input_audio_buffer.speech_stopped`, `input_audio_buffer.committed`, `response.created`, `response.audio_transcript.delta`, `response.audio.delta`, `response.audio.done`, `response.done`, `error`