README: document Plan 2 device-hub surface area
This commit is contained in:
@@ -4,8 +4,12 @@ Voice assistant on a Raspberry Pi talking to the OpenAI Realtime API via an
|
|||||||
ASP.NET backend deployed on Coolify.
|
ASP.NET backend deployed on Coolify.
|
||||||
|
|
||||||
See `docs/superpowers/specs/2026-06-11-smart-assistant-design.md` for the full
|
See `docs/superpowers/specs/2026-06-11-smart-assistant-design.md` for the full
|
||||||
design and `docs/superpowers/plans/2026-06-11-smart-assistant-backend-foundation.md`
|
design. Implementation plans:
|
||||||
for the Plan 1 implementation steps that produced this code.
|
|
||||||
|
- Plan 1 — `docs/superpowers/plans/2026-06-11-smart-assistant-backend-foundation.md`
|
||||||
|
(Identity, devices, pairing, install endpoints).
|
||||||
|
- Plan 2 — `docs/superpowers/plans/2026-06-11-smart-assistant-device-hub-realtime-tools.md`
|
||||||
|
(device WebSocket hub, OpenAI Realtime relay, tools registry).
|
||||||
|
|
||||||
## Repo layout
|
## Repo layout
|
||||||
|
|
||||||
@@ -84,11 +88,59 @@ curl -sf -X POST "$COOLIFY_URL/api/v1/deploy?uuid=$APP" \
|
|||||||
Fire-and-forget; the API returns a `deployment_uuid` immediately. Don't poll
|
Fire-and-forget; the API returns a `deployment_uuid` immediately. Don't poll
|
||||||
(per the global Coolify rule in `CLAUDE.md`).
|
(per the global Coolify rule in `CLAUDE.md`).
|
||||||
|
|
||||||
|
## Plan 2: device hub + Realtime relay
|
||||||
|
|
||||||
|
- `wss://<domain>/device` — bearer-token WebSocket. Pair a device first to get a token.
|
||||||
|
- Handshake: client sends `{type:"hello", device_id, client_version}`, backend
|
||||||
|
replies `hello_ack` with per-device config (voice, model, system prompt, idle
|
||||||
|
timeout, enabled tools).
|
||||||
|
- Heartbeat: client sends `{type:"ping"}` every 15 s, backend replies `pong`
|
||||||
|
and updates `Devices.LastSeenAt`.
|
||||||
|
- Wake: `{type:"wake"}` opens a Realtime session against OpenAI; backend emits
|
||||||
|
`session_started` (with `conversation_id`), forwards `response.audio.delta`
|
||||||
|
as binary frames to the device, persists `response.audio_transcript.done` as
|
||||||
|
an assistant turn, persists `conversation.item.input_audio_transcription.completed`
|
||||||
|
as a user turn, and emits `assistant_done` after each `response.done`.
|
||||||
|
- Tools shipped (per-device toggle via `DeviceConfig.EnabledToolsJson`):
|
||||||
|
- `get_current_time` (server-side, returns `{"now":"<ISO-UTC>"}`),
|
||||||
|
- `end_session` (closes the session after the current `response.done`
|
||||||
|
completes; emits `session_ended(reason="tool")`),
|
||||||
|
- `set_volume` (round-trips a `tool_call`/`tool_result` envelope to the Pi;
|
||||||
|
the Pi side will run `amixer` in Plan 3).
|
||||||
|
- Idle timeout: server-side. With no `input_audio_buffer.speech_started` upstream
|
||||||
|
event for `IdleTimeoutSeconds`, the relay emits `session_ended(reason="idle")`.
|
||||||
|
|
||||||
|
The Python Pi client (Plan 3) is the production consumer of this surface. A
|
||||||
|
quick handshake-only smoke against the live deploy:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
DOMAIN="https://assistant.volcanic.tes.gd"
|
||||||
|
# Reuse the curl recipe above to log in, generate a pair-code, and pair
|
||||||
|
# a "smoke" device — capture $TOKEN from the /api/pair response.
|
||||||
|
python3 - <<PY
|
||||||
|
import asyncio, json, websockets
|
||||||
|
URL, TOKEN = "wss://assistant.volcanic.tes.gd/device", "$TOKEN"
|
||||||
|
async def main():
|
||||||
|
async with websockets.connect(
|
||||||
|
URL, additional_headers={"Authorization": f"Bearer {TOKEN}"}
|
||||||
|
) as ws:
|
||||||
|
await ws.send(json.dumps({
|
||||||
|
"type":"hello","device_id":"00000000-0000-0000-0000-000000000000",
|
||||||
|
"client_version":"0.1"}))
|
||||||
|
print("ack:", json.loads(await ws.recv())["type"])
|
||||||
|
await ws.send(json.dumps({"type":"ping"}))
|
||||||
|
print("pong:", json.loads(await ws.recv())["type"])
|
||||||
|
asyncio.run(main())
|
||||||
|
PY
|
||||||
|
```
|
||||||
|
|
||||||
|
Don't send `wake` from the smoke — it would open a real OpenAI session and
|
||||||
|
burn budget. Plan 3 exercises that end-to-end.
|
||||||
|
|
||||||
## What's next
|
## What's next
|
||||||
|
|
||||||
- **Plan 2** — backend device hub + OpenAI Realtime relay + tools registry
|
|
||||||
(the `end_session`, `get_current_time`, `set_volume` tools the spec describes).
|
|
||||||
- **Plan 3** — full Python client on the Pi (wakeword, VAD, audio plumbing,
|
- **Plan 3** — full Python client on the Pi (wakeword, VAD, audio plumbing,
|
||||||
state machine, install + pair CLIs).
|
state machine, install + pair CLIs). This is the production caller of
|
||||||
- **Plan 4** — React management UI for the admin dashboard and conversation
|
`/device` and the integration test for everything above.
|
||||||
history viewer.
|
- **Plan 4** — React management UI for the admin dashboard, per-device
|
||||||
|
config editor, and conversation history viewer.
|
||||||
|
|||||||
Reference in New Issue
Block a user