Hardware verification surfaced four shape changes from the beta API that
the spec/plan was written against:
1. Remove the OpenAI-Beta: realtime=v1 header (GA rejects it with
beta_api_shape_disabled).
2. session.update payload nests under session.audio.{input,output} instead
of flat fields. session.type = "realtime" + session.model are inline.
3. session.audio.{input,output}.format is now an object
({ type: "audio/pcm", rate: 24000 }), not a bare "pcm16" string.
4. 'session.modalities' is no longer accepted; dropped (server uses defaults).
5. response.audio.* → response.output_audio.* (delta, done, transcript).
Also silenced a bunch of GA framing events (conversation.item.added /
.done, response.output_item.added / .done, response.content_part.added /
.done, rate_limits.updated) that were burying signal in the [ws] ignored
log lines.
Plus added an outer try/catch around main so a mid-startup WS error event
(which fires cts.Cancel via the receive loop) routes through cleanup
instead of crashing on the await SendSessionUpdate.
Verified on hardware: full round-trip works, t=0.44 s end-of-speech to
first audio sample arriving, assistant reply audible, clean exit.
Code-review feedback from Task 2:
- Use state.NoMoreDeltas in the output callback completion check instead
of downstream.Reader.Completion.IsCompleted (spec called for it as the
redundant fast path; it was dead code before).
- UpstreamSendLoop now break's when StopSending && upstream channel empty,
matching the spec's stated exit shape.
Five tasks: scaffold csproj → write full Program.cs (WS + audio
streams + channels + server VAD round-trip) → bin/probe-cs-4a with
~/.openai_key precondition → one-time Pi key setup → hardware
verification against the spec's 8 pass criteria.
Boot → beep → user speaks → server VAD ends turn → assistant audio
plays back → exit 0. No wakeword, no loop, no state machine.
Splits Test 4 into 4a (this spec) and 4b (wakeword + loop + 16↔24 kHz
coexistence, future). 4a debugs the OpenAI Realtime surface — auth, WS
handshake, session.update payload, server VAD, base64 audio, streaming-
length output-stream cleanup — in isolation from 4b's harder concurrency.
FireAndForgetPlayback's cleanup task had the same 2 s done.Wait timeout
as the beep cleanup. Beep is 200 ms so 2 s is generous; recording
playback is 5 s so the timeout fired mid-playback, cleanup ran
stream.Stop(), and the audio cut at ~2.2 s. Timeout is now
recording-duration + 2 s.
Stale Set() from a delayed cleanup task could short-circuit a future
PLAYBACK state; new flag per cycle avoids the cross-cycle leak. Also
log when the 2 s done.Wait timeout fires so silent PortAudio callback
faults are visible during hardware debugging.
Five tasks: scaffold csproj, copy WakewordModel + models with Reset(),
write Program.cs with the four-state machine, write bin/probe-cs-3
deploy script, and hardware verification on the Pi.
State machine on a single persistent input stream: IDLE → BEEPING →
RECORDING → PLAYBACK → IDLE. Detector gated during non-IDLE states;
WakewordModel.Reset() called on IDLE re-entry to clear stale ring
buffers and re-arm the warmup guard.
Add PrintShapes() helper that writes input/output name, shape, and dtype
for each InferenceSession to stderr at construction time, prefixed with
[wakeword-model]. Called after each of the three session creations so the
geometry is visible on first run without grep-ing logs.
Also rephrase the stale "FIXED:" comment on the mel Dimensions[2] indexing
to plain explanatory text that describes the current invariant.
Replaces the Task 3 smoke-test Program.cs with the full inference loop:
- PA_ALSA_PLUGHW=1 and ORT_LOGGING_LEVEL=3 via both Environment.SetEnvironmentVariable
and Libc.setenv (P/Invoke) per the Test 1 findings.md env-var gotcha.
- OrtEnv.CreateInstanceWithOptions with ORT_LOGGING_LEVEL_ERROR to suppress
early GPU-discovery warnings from device_discovery.cc in ORT 1.26.0
(the env var alone does not silence these; CreateInstanceWithOptions does).
- Callback-driven PortAudio input stream at 16 kHz / Int16 / 1280 frames/block.
- BlockingCollection<short[]>(16) queue; per-call new short[1280] allocation (no races).
- Main loop: Take() -> WakewordModel.Predict() -> threshold >= 0.5 with 1.0s cooldown
-> DETECTED alexa score=... t=...s. No beep (Task 5 adds it).
- Console.CancelKeyPress -> cts.Cancel() -> graceful stream.Stop() + model.Dispose()
+ PortAudio.Terminate().
Ports openwakeword 0.6.0's AudioFeatures._streaming_features() and
Model.predict() to C# / Microsoft.ML.OnnxRuntime 1.26.0.
Bug fixed vs plan: actual melspectrogram.onnx output shape is
(1, 1, n_frames, 32) — n_frames lives at Dimensions[2], not [1].
Access pattern corrected to melOut[0, 0, f, b].
Smoke test on Pi: 25/50 non-zero silence scores, max ≈ 0.000064
(matches Python reference: 45/50 non-zero, same magnitude).
"Predict pipeline ran without throwing." confirmed.
Models sourced from openwakeword==0.6.0 on the Pi (alexa_v0.1.onnx
vendored as alexa.onnx). ORT 1.26.0 loads all three on linux-arm64
self-contained publish; libonnxruntime.so confirmed in publish dir.
SHA-256:
melspectrogram.onnx ba2b0e0f8b7b875369a2c89cb13360ff53bac436f2895cced9f479fa65eb176f
embedding_model.onnx 70d164290c1d095d1d4ee149bc5e00543250a7316b59f31d056cff7bd3075c1f
alexa.onnx 6ff566a01d12670e8d9e3c59da32651db1575d17272a601b7f8a39283dfbae3e
tes