Findings: Test 4a OpenAI Realtime one-shot voice round-trip outcome

findings.md

@@ -244,6 +244,75 @@ Test 3 stitched the C# Test 1 (record + play) and Test 2 (wakeword + beep) into
 - Code: `tests/03-full-cycle-cs/`
 - Deploy: `bin/probe-cs-3`
 
+## Test 4a outcome (2026-06-12 — OpenAI Realtime one-shot voice round-trip in C#)
+
+Test 4a opens a WebSocket to `wss://api.openai.com/v1/realtime?model=gpt-realtime`, plays a startup beep, streams mic audio to the server VAD endpoint, plays the assistant's audio reply back through the same USB Speaker Phone, and exits. Probe lives at `tests/04a-realtime-oneshot-cs/`; deploy script `bin/probe-cs-4a`. All 8 pass criteria met on hardware after a four-round iteration to match the GA Realtime API surface (the spec was written against beta-era docs from openai/openai-realtime-api-beta).
+
+### What we proved
+
+- **`ClientWebSocket` + two background `Task.Run` loops (ReceiveLoop, UpstreamSendLoop) + bounded `Channel<short[]>` on each direction** is the right shape on .NET 9 / linux-arm64 for a Realtime API client. No third-party WebSocket lib needed; `System.Text.Json` is fine for event parsing via `JsonDocument`.
+- **One PortAudio input stream + one persistent output stream at 24 kHz mono Int16, both on the USB Speaker Phone, coexist cleanly.** PortAudio plug-resamples 48↔24 at ALSA on this device. No client-side resampling.
+- **Server VAD endpointing works out of the box.** `turn_detection: { type: "server_vad", threshold: 0.5, prefix_padding_ms: 300, silence_duration_ms: 500, create_response: true }` (nested under `session.audio.input`) fires `speech_started` / `speech_stopped` correctly on real Pi mic input; server auto-commits the buffer and creates the response.
+- **Round-trip latency is excellent.** End-of-speech (server VAD `speech_stopped`) to first response audio sample arriving at the probe: **0.44 s** measured on the test "hello" turn. Spec target was under 2 s; we beat it by 4×.
+- **Streaming-length cleanup discipline works.** Output callback signals `doneFlag` when `CurrentChunk == null && Reader.Count == 0 && NoMoreDeltas` (the `NoMoreDeltas` `volatile bool` is set on `response.done` after `Writer.Complete()`; reading it is cheaper than touching `Reader.Completion`). Probe exits ~immediately after `response.done`.
+- **`~/.openai_key` precondition pattern in `bin/probe-cs-4a` works.** Pre-publish check via `sshpass ... 'test -f ~/.openai_key && test -r ~/.openai_key'` catches a missing key at the deploy step rather than 30 s into publish + scp. Set with `umask 077 && cat > ~/.openai_key <<KEY` on the Pi.
+
+### Key gotchas — Realtime API GA migration
+
+The spec was written against the older beta API surface (the openai/openai-realtime-api-beta library docs, which were the largest indexed source). The GA API rejected every beta-era piece of our session.update payload, surfacing one error per iteration. The full diff (commit `4cfb0d4`):
+
+1. **No `OpenAI-Beta: realtime=v1` request header.** GA rejects it as `beta_api_shape_disabled`. Only `Authorization: Bearer <key>` is required for GA.
+2. **`session.update` payload is nested under `session.audio.{input,output}` instead of flat at the top of `session`.** Beta had `session.input_audio_format`, `session.output_audio_format`, `session.voice`, `session.turn_detection`. GA has `session.audio.input.format`, `session.audio.input.turn_detection`, `session.audio.output.format`, `session.audio.output.voice`. Also `session.type = "realtime"` and `session.model = "gpt-realtime"` go inline at the top of `session`.
+3. **`session.audio.{input,output}.format` is now an OBJECT, not a bare string.** Beta accepted `"pcm16"`. GA requires `{ "type": "audio/pcm", "rate": 24000 }`. The error was `Invalid type for 'session.audio.input.format': expected an object, but got a string instead.`
+4. **`session.modalities` is no longer accepted.** Server rejected with `Unknown parameter: 'session.modalities'`. Dropped from our payload; server defaults are fine (audio + text output). The GA equivalent for constraining output is likely `output_modalities` (matches Responses API convention) — not needed for the probe.
+5. **Server events for response audio are renamed `response.audio.*` → `response.output_audio.*`.** Beta had `response.audio.delta` / `.done` / `response.audio_transcript.delta` / `.done`. GA has `response.output_audio.delta` / `.done` / `response.output_audio_transcript.delta` / `.done`. **This is the silent failure mode**: the receive loop's switch default branch quietly logs `[ws] ignored type=…` instead of writing the decoded audio to the output channel. The probe logs "session pinned", VAD fires, response completes, transcript is empty, no audio is heard. Took one round of hardware verification to surface; trivially diagnosed via the `ignored type=…` log lines.
+6. **GA emits a stream of conversation/response framing events** that don't exist in beta: `conversation.item.added` / `.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `rate_limits.updated`. Logged as `[ws] ignored type=…` initially; now silenced in our switch so they don't bury real signal.
+
+The current Realtime model name `gpt-realtime` is still accepted (the dev docs mention `gpt-realtime-2` as well; we didn't need to swap).
+
+### Other gotchas
+
+- **Mid-startup WS error must route through cleanup.** Initial implementation had a single `try { … } finally { … }` around main. When the receive loop processed an error event and called `cts.Cancel()` mid-`SendSessionUpdate`, the `await ws.SendAsync(…, cts.Token)` threw `TaskCanceledException`, which propagated past the finally as an unhandled exception (PortAudio + ws Dispose still ran from finally, but the process exited non-zero with a stack trace). Fix: add `catch (OperationCanceledException) { /* expected on cancel */ } catch (Exception ex) { log; ExitCode = 7 }` between the outer try and finally. Probe now logs the `[error] WS error event:` from the receive loop and exits cleanly. Commit `4cfb0d4`.
+- **`Environment.Exit(N)` skips the outer `finally` and leaks PortAudio**, but is OK in the LoadApiKey path because it runs before `PortAudio.Initialize()`. The `Environment.ExitCode = N; return;` pattern in WS-connect / stream-Start failure paths preserves the finally correctly.
+- **CA1416 warning on `File.GetUnixFileMode`** is benign; the call is guarded by `try { … } catch (PlatformNotSupportedException) { }`. The build emits one warning consistently; no action needed.
+- **ALSA shutdown noise on first run.** Two `alsa_snd_pcm_drop` "failed" lines appeared from PortAudio's cleanup on the *failed* runs (cts.Cancel from an error event tore down both streams mid-init). Did NOT appear on the successful run. Looks like a PortAudio cleanup-on-abort quirk; only present on error paths.
+
+### Pass criteria
+
+| # | Criterion | Result |
+|-|-|-|
+| 1 | End-to-end round-trip works once | ✅ Probe boots, beeps, user speaks, audible reply, exits 0 |
+| 2 | Response intelligible | ✅ Clean speech, no clipping, no dropouts |
+| 3 | Latency < 2 s | ✅ **0.44 s** end-of-speech to first delta |
+| 4 | No `[status] upstream behind` | ✅ None |
+| 5 | No `[error] WS …` during normal flow | ✅ None |
+| 6 | No PortAudio `InputOverflow` | ✅ None |
+| 7 | Ctrl-C clean exit | ✅ Same cleanup path as the successful run, which exited cleanly |
+| 8 | Exits within ~500 ms of last audio sample | ✅ `response.done` → `[ws] closing` → `bye` is instant |
+
+### Implications for Test 4b
+
+- **GA API surface is now proven.** Test 4b inherits the migrated `SendSessionUpdate` payload, the renamed receive-loop switch cases, the silenced GA framing events, and the outer try/catch shape. Lift them directly.
+- **The "spec written against beta docs" failure mode is a real risk for any future Realtime API work.** When LLM-fetched docs (context7, training data) disagree with the live server, **only the server is authoritative**. Pattern: send a minimal session.update, observe the error response, iterate. Our four-round iteration took ~10 min total.
+- **The "[ws] ignored type=…" log line on the default branch is the diagnostic** that surfaced the renames. Keep it (or an equivalent) in 4b's receive loop. Without it, missing audio could have been silently attributed to a thousand other causes.
+- **Server VAD parameters (`threshold: 0.5`, `silence_duration_ms: 500`, `prefix_padding_ms: 300`) work well on this hardware** — natural-feeling end-of-turn after ~500 ms of silence, no clipped openings of user speech. Lift unchanged into 4b.
+- **`Environment.Exit` vs `Environment.ExitCode = N; return;`** — the latter is the correct pattern for failure paths that need PortAudio + WS cleanup. 4b will have more failure paths (wakeword model init, audio rate conversion, etc.); use the ExitCode pattern consistently.
+- **The startup beep pattern (queue beep onto the persistent output channel before `_micArmed = true`)** works — beep does not get captured and shipped to OpenAI. Carry into 4b.
+- **The 16↔24 kHz coexistence question is the one major thing 4a did not address.** 4a is 24 kHz end-to-end. 4b must deal with the wakeword model requiring 16 kHz input while OpenAI requires 24 kHz pcm16 — either a second input stream, an in-process resampler, or open one stream at 24 kHz and downsample for the wakeword. The spec/plan for 4b will need to pick.
+
+### Reference paths (Test 4a)
+
+- Spec: `docs/superpowers/specs/2026-06-12-test-4a-openai-realtime-design.md`
+- Plan: `docs/superpowers/plans/2026-06-12-test-4a-openai-realtime.md`
+- Code: `tests/04a-realtime-oneshot-cs/`
+- Deploy: `bin/probe-cs-4a`
+- GA Realtime API headers / URL / session.update shape (as discovered against the live server, 2026-06-12):
+  - `wss://api.openai.com/v1/realtime?model=gpt-realtime`
+  - Header `Authorization: Bearer <key>` (NO `OpenAI-Beta`)
+  - `session.update` payload: `{ type: "session.update", session: { type: "realtime", model: "gpt-realtime", instructions: "...", audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: { type: "server_vad", threshold, prefix_padding_ms, silence_duration_ms, create_response } }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "alloy" } } } }`
+  - Server events to handle: `session.created`, `session.updated`, `input_audio_buffer.speech_started`, `input_audio_buffer.speech_stopped`, `input_audio_buffer.committed`, `response.created`, `response.output_audio_transcript.delta`, `response.output_audio.delta`, `response.output_audio.done`, `response.output_audio_transcript.done`, `response.done`, `error`
+  - Server events safe to ignore for a probe: `conversation.item.added`, `conversation.item.done`, `response.output_item.added`, `response.output_item.done`, `response.content_part.added`, `response.content_part.done`, `rate_limits.updated`
+
 ## Reference paths
 
 - Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`