Assistant/docs/superpowers/plans/2026-06-11-smart-assistant-device-hub-realtime-tools.md

Smart Assistant — Device Hub + Realtime Relay + Tools Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stand up the per-device WebSocket hub at /device, the OpenAI Realtime relay (one upstream WSS per conversation), the in-process C# tools registry (get_current_time, end_session, set_volume), and the Conversations / Turns persistence — enough that a paired client can wake, hold a multi-turn conversation against OpenAI, end via tool-call or idle timeout, and have every utterance + tool call written to SQLite. The Python client itself is Plan 3 — this plan ships with a minimal wscat-like manual smoke script and integration tests against a scripted upstream double.

Architecture: A single persistent client WS per paired device, authenticated by the bearer device token. The handler (DeviceHubEndpoint) routes text envelopes (hello/ping/wake/cancel/playback_done/tool_result) and binary frames (PCM16 uplink). On wake, the hub creates a RealtimeSession that opens its own upstream WSS to OpenAI (/v1/realtime?model=…), drives a session.update with per-device config + tools, and pumps events both ways. The relay is layered behind IRealtimeUpstream so tests inject a scripted double — no real OpenAI calls in unit tests. Tools implement ITool and run in-process; set_volume routes back to the Pi via IDeviceChannel.CallPiToolAsync(...). Idle timeout, end_session sequencing, and transcript persistence are all part of RealtimeSession.

Tech Stack: .NET 9 ASP.NET Core (WebSockets middleware, System.Net.WebSockets.ClientWebSocket for upstream), System.Text.Json with JsonNamingPolicy.SnakeCaseLower, EF Core 9 + SQLite (already in Plan 1), System.Threading.Channels for queues, xUnit + Microsoft.AspNetCore.Mvc.Testing for integration tests.

Scope (in):

• Conversation + Turn entities + migration.
• ConversationLog (write-only persistence helper).
• ITool + ToolRegistry + three built-ins: get_current_time, end_session, set_volume.
• OpenAIKeyProvider.
• IRealtimeUpstream abstraction + ScriptedRealtimeUpstream test double + real OpenAIRealtimeUpstream.
• RealtimeSession (open → session.update → pump → tools → idle/end → close, with transcript writes).
• ActiveDevice + IDeviceChannel + DeviceRegistry.
• DeviceHubEndpoint at /device with bearer-token auth, handshake, ping, wake, cancel, playback_done, tool_result.
• Full integration test (real WS in WebApplicationFactory + scripted upstream).
• Coolify deploy + smoke check on the live /device endpoint.

Scope (out, later plans):

• Python client implementation (Plan 3 — replaces the placeholders shipped in Plan 1 with real audio loop, wakeword, state machine, playback, pairing CLI).
• Admin UI / DeviceConfig edit endpoints / config_updated push from a UI save (Plan 4 — the relay still consumes DeviceConfig from the DB on each wake, so changing rows directly works today).
• play_audio(kind=notification) (backend pushing audio outside a session) — envelope types are defined here but no production flow uses them yet.
• Long-term memory, audio recording persistence, barge-in (all non-goals per spec).
• React UI (Plan 4).

Reference spec: docs/superpowers/specs/2026-06-11-smart-assistant-design.md Previous plan: docs/superpowers/plans/2026-06-11-smart-assistant-backend-foundation.md

---

File structure

This plan creates the following files. Each file owns one concern; cross-references are noted under each task.

Assistant/
├── backend/
│   ├── Conversations/
│   │   ├── Conversation.cs                  # entity
│   │   ├── Turn.cs                          # entity
│   │   └── ConversationLog.cs               # scoped writer
│   ├── Tools/
│   │   ├── ITool.cs                         # interface, DeviceContext, ToolInvocation, ToolResult
│   │   ├── IDeviceChannel.cs                # surface tools see to call Pi-side tools
│   │   ├── ToolRegistry.cs                  # singleton with all ITool impls
│   │   ├── GetCurrentTimeTool.cs            # server-side, no params
│   │   ├── EndSessionTool.cs                # marks RealtimeSession to close after response.done
│   │   └── SetVolumeTool.cs                 # routes to Pi via IDeviceChannel
│   ├── Realtime/
│   │   ├── OpenAIKeyProvider.cs             # reads OPENAI_API_KEY env
│   │   ├── RealtimeSettings.cs              # per-conversation config (model/voice/prompt/tools/timeout)
│   │   ├── RealtimeEvents.cs                # records for upstream JSON envelopes (session.update etc.)
│   │   ├── IRealtimeUpstream.cs             # abstraction: send JSON, receive JSON, close
│   │   ├── OpenAIRealtimeUpstream.cs        # real ClientWebSocket implementation
│   │   ├── RealtimeSession.cs               # owns one conversation; pumps both ways
│   │   └── RealtimeSessionFactory.cs        # creates RealtimeSession with DI'd deps
│   └── DeviceHub/
│       ├── HubEnvelopes.cs                  # records for client-side wire types
│       ├── ActiveDevice.cs                  # per-WS state; implements IDeviceChannel; outbound writer
│       ├── DeviceRegistry.cs                # singleton map deviceId→ActiveDevice
│       ├── DeviceAuth.cs                    # bearer-token → device lookup helper
│       └── DeviceHubEndpoint.cs             # /device WS handler
├── backend.tests/
│   ├── ConversationLogTests.cs
│   ├── ToolRegistryTests.cs
│   ├── GetCurrentTimeToolTests.cs
│   ├── EndSessionToolTests.cs
│   ├── SetVolumeToolTests.cs
│   ├── ScriptedRealtimeUpstream.cs          # test double
│   ├── FakeDeviceChannel.cs                 # test double
│   ├── RealtimeSessionLifecycleTests.cs
│   ├── RealtimeSessionAudioTests.cs
│   ├── RealtimeSessionTranscriptTests.cs
│   ├── RealtimeSessionToolsTests.cs
│   ├── RealtimeSessionEndSessionTests.cs
│   ├── RealtimeSessionIdleTimeoutTests.cs
│   ├── DeviceRegistryTests.cs
│   ├── DeviceHubHandshakeTests.cs
│   └── DeviceHubWakeFlowTests.cs

---

Task 1: Conversation + Turn entities + migration

Files:

• Create: backend/Conversations/Conversation.cs

• Create: backend/Conversations/Turn.cs

• Modify: backend/Data/AppDbContext.cs

• Step 1: Define Conversation

Create backend/Conversations/Conversation.cs:

namespace backend.Conversations;

public enum EndReason
{
    Unknown = 0,
    Tool = 1,
    Idle = 2,
    Error = 3,
}

public class Conversation
{
    public Guid Id { get; set; } = Guid.NewGuid();
    public Guid DeviceId { get; set; }
    public DateTimeOffset StartedAt { get; set; }
    public DateTimeOffset? EndedAt { get; set; }
    public EndReason EndReason { get; set; } = EndReason.Unknown;
    public List<Turn> Turns { get; set; } = new();
}

• Step 2: Define Turn

Create backend/Conversations/Turn.cs:

namespace backend.Conversations;

public enum TurnRole
{
    User = 0,
    Assistant = 1,
    Tool = 2,
}

public class Turn
{
    public Guid Id { get; set; } = Guid.NewGuid();
    public Guid ConversationId { get; set; }
    public int Index { get; set; }
    public TurnRole Role { get; set; }
    public string? Text { get; set; }
    public string? ToolName { get; set; }
    public string? ToolArgsJson { get; set; }
    public string? ToolResultJson { get; set; }
    public DateTimeOffset CreatedAt { get; set; }
}

• Step 3: Register entities in AppDbContext

In backend/Data/AppDbContext.cs, add using backend.Conversations; at the top, then add the DbSets and the mapping in OnModelCreating:

public DbSet<Conversation> Conversations => Set<Conversation>();
public DbSet<Turn> Turns => Set<Turn>();

Inside OnModelCreating (after the existing mappings, before the closing brace):

b.Entity<Conversation>().HasIndex(c => c.DeviceId);
b.Entity<Conversation>()
    .HasMany(c => c.Turns)
    .WithOne()
    .HasForeignKey(t => t.ConversationId)
    .OnDelete(DeleteBehavior.Cascade);
b.Entity<Turn>().HasIndex(t => new { t.ConversationId, t.Index }).IsUnique();

• Step 4: Generate the migration

dotnet ef migrations add AddConversationsAndTurns --project backend --startup-project backend

Expected: a new file under backend/Migrations/ ending in _AddConversationsAndTurns.cs that creates Conversations and Turns tables.

• Step 5: Verify build + previous tests still pass

Run: dotnet test Expected: every Plan 1 test still passes; no new failures.

• Step 6: Commit

git add backend/Conversations backend/Data/AppDbContext.cs backend/Migrations
git commit -m "Add Conversation + Turn entities and migration"

---

Task 2: ConversationLog (scoped persistence helper)

Files:

• Create: backend/Conversations/ConversationLog.cs

• Create: backend.tests/ConversationLogTests.cs

• Modify: backend/Program.cs

• Step 1: Write the failing test

Create backend.tests/ConversationLogTests.cs:

using backend.Conversations;
using backend.Data;
using FluentAssertions;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class ConversationLogTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    private async Task<(AppDbContext db, ConversationLog log, Guid deviceId)> ArrangeAsync()
    {
        // Use the factory's DI container so we hit the same SQLite file.
        var scope = _factory.Services.CreateScope();
        var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var deviceId = Guid.NewGuid();
        return (db, log, deviceId);
    }

    [Fact]
    public async Task Start_creates_conversation_row()
    {
        var (db, log, deviceId) = await ArrangeAsync();

        var convo = await log.StartAsync(deviceId, default);

        var row = await db.Conversations.FirstAsync(c => c.Id == convo.Id);
        row.DeviceId.Should().Be(deviceId);
        row.EndedAt.Should().BeNull();
        row.EndReason.Should().Be(EndReason.Unknown);
    }

    [Fact]
    public async Task Turns_are_persisted_with_monotonic_index_per_conversation()
    {
        var (db, log, deviceId) = await ArrangeAsync();
        var convo = await log.StartAsync(deviceId, default);

        await log.AppendUserAsync(convo.Id, "hi", default);
        await log.AppendAssistantAsync(convo.Id, "hello there", default);
        await log.AppendToolAsync(convo.Id, "get_current_time", "{}", "{\"now\":\"2026-06-11T00:00:00Z\"}", default);

        var turns = await db.Turns
            .Where(t => t.ConversationId == convo.Id)
            .OrderBy(t => t.Index)
            .ToListAsync();

        turns.Should().HaveCount(3);
        turns.Select(t => t.Index).Should().Equal(0, 1, 2);
        turns.Select(t => t.Role).Should().Equal(TurnRole.User, TurnRole.Assistant, TurnRole.Tool);
        turns[0].Text.Should().Be("hi");
        turns[1].Text.Should().Be("hello there");
        turns[2].ToolName.Should().Be("get_current_time");
    }

    [Fact]
    public async Task End_sets_ended_at_and_reason()
    {
        var (db, log, deviceId) = await ArrangeAsync();
        var convo = await log.StartAsync(deviceId, default);

        await log.EndAsync(convo.Id, EndReason.Idle, default);

        var row = await db.Conversations.FirstAsync(c => c.Id == convo.Id);
        row.EndedAt.Should().NotBeNull();
        row.EndReason.Should().Be(EndReason.Idle);
    }
}

• Step 2: Run the test to see it fail

Run: dotnet test --filter ConversationLogTests Expected: build error (ConversationLog not defined).

• Step 3: Implement the writer

Create backend/Conversations/ConversationLog.cs:

using backend.Data;
using Microsoft.EntityFrameworkCore;

namespace backend.Conversations;

public class ConversationLog(AppDbContext db)
{
    public async Task<Conversation> StartAsync(Guid deviceId, CancellationToken ct)
    {
        var c = new Conversation
        {
            DeviceId = deviceId,
            StartedAt = DateTimeOffset.UtcNow,
        };
        db.Conversations.Add(c);
        await db.SaveChangesAsync(ct);
        return c;
    }

    public Task AppendUserAsync(Guid conversationId, string text, CancellationToken ct)
        => AppendAsync(conversationId, TurnRole.User, text: text, ct: ct);

    public Task AppendAssistantAsync(Guid conversationId, string text, CancellationToken ct)
        => AppendAsync(conversationId, TurnRole.Assistant, text: text, ct: ct);

    public Task AppendToolAsync(
        Guid conversationId, string toolName, string argsJson, string resultJson, CancellationToken ct)
        => AppendAsync(conversationId, TurnRole.Tool,
            toolName: toolName, argsJson: argsJson, resultJson: resultJson, ct: ct);

    public async Task EndAsync(Guid conversationId, EndReason reason, CancellationToken ct)
    {
        var row = await db.Conversations.FirstAsync(c => c.Id == conversationId, ct);
        row.EndedAt = DateTimeOffset.UtcNow;
        row.EndReason = reason;
        await db.SaveChangesAsync(ct);
    }

    private async Task AppendAsync(
        Guid conversationId, TurnRole role,
        string? text = null, string? toolName = null,
        string? argsJson = null, string? resultJson = null,
        CancellationToken ct = default)
    {
        var next = await db.Turns
            .Where(t => t.ConversationId == conversationId)
            .Select(t => (int?)t.Index)
            .MaxAsync(ct) ?? -1;

        db.Turns.Add(new Turn
        {
            ConversationId = conversationId,
            Index = next + 1,
            Role = role,
            Text = text,
            ToolName = toolName,
            ToolArgsJson = argsJson,
            ToolResultJson = resultJson,
            CreatedAt = DateTimeOffset.UtcNow,
        });
        await db.SaveChangesAsync(ct);
    }
}

• Step 4: Register in DI

In backend/Program.cs, in the existing service-registration block (above var app = builder.Build();), add:

builder.Services.AddScoped<backend.Conversations.ConversationLog>();

• Step 5: Run the tests

Run: dotnet test --filter ConversationLogTests Expected: Passed: 3.

• Step 6: Commit

git add backend/Conversations/ConversationLog.cs backend.tests/ConversationLogTests.cs backend/Program.cs
git commit -m "Add ConversationLog (start/append/end with monotonic per-convo index)"

---

Task 3: ITool, IDeviceChannel, ToolRegistry, GetCurrentTimeTool

Files:

• Create: backend/Tools/ITool.cs

• Create: backend/Tools/IDeviceChannel.cs

• Create: backend/Tools/ToolRegistry.cs

• Create: backend/Tools/GetCurrentTimeTool.cs

• Create: backend.tests/ToolRegistryTests.cs

• Create: backend.tests/GetCurrentTimeToolTests.cs

• Step 1: Write the failing tests

Create backend.tests/ToolRegistryTests.cs:

using backend.Tools;
using FluentAssertions;
using Xunit;

namespace backend.tests;

public class ToolRegistryTests
{
    private readonly ITool[] _all = new ITool[]
    {
        new GetCurrentTimeTool(),
    };

    [Fact]
    public void Get_returns_tool_by_exact_name()
    {
        var reg = new ToolRegistry(_all);
        reg.Get("get_current_time").Should().NotBeNull();
        reg.Get("does_not_exist").Should().BeNull();
    }

    [Fact]
    public void Filter_returns_only_enabled_tools_preserving_order()
    {
        var reg = new ToolRegistry(_all);
        var enabled = new HashSet<string> { "get_current_time", "ghost" };
        reg.EnabledFor(enabled).Select(t => t.Name).Should().Equal("get_current_time");
    }
}

Create backend.tests/GetCurrentTimeToolTests.cs:

using System.Text.Json;
using backend.Tools;
using FluentAssertions;
using Xunit;

namespace backend.tests;

public class GetCurrentTimeToolTests
{
    [Fact]
    public async Task Returns_now_field_with_iso_utc_timestamp()
    {
        var tool = new GetCurrentTimeTool();
        var args = JsonDocument.Parse("{}").RootElement;
        var ctx = new DeviceContext(Guid.NewGuid(), Guid.NewGuid(), new NoopChannel());

        var res = await tool.ExecuteAsync(new ToolInvocation("call_1", args), ctx, default);

        res.Output.TryGetProperty("now", out var now).Should().BeTrue();
        now.GetString().Should().MatchRegex(@"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}");
        now.GetString()!.Should().EndWith("Z");
    }

    private class NoopChannel : IDeviceChannel
    {
        public Task<JsonElement> CallPiToolAsync(string name, JsonElement args, CancellationToken ct)
            => throw new NotSupportedException();
    }
}

• Step 2: Run to see them fail

Run: dotnet test --filter "ToolRegistryTests|GetCurrentTimeToolTests" Expected: build error (ITool, ToolRegistry, GetCurrentTimeTool, IDeviceChannel not defined).

• Step 3: Define the tool interface

Create backend/Tools/ITool.cs:

using System.Text.Json;

namespace backend.Tools;

public record DeviceContext(Guid DeviceId, Guid ConversationId, IDeviceChannel Channel);
public record ToolInvocation(string CallId, JsonElement Arguments);
public record ToolResult(JsonElement Output);

public interface ITool
{
    string Name { get; }
    string Description { get; }
    JsonElement ParameterSchema { get; }
    bool RunsDuringResponse { get; } // false → executes at response.done; end_session uses this
    Task<ToolResult> ExecuteAsync(
        ToolInvocation invocation,
        DeviceContext device,
        CancellationToken ct);
}

• Step 4: Define IDeviceChannel

Create backend/Tools/IDeviceChannel.cs:

using System.Text.Json;

namespace backend.Tools;

public interface IDeviceChannel
{
    Task<JsonElement> CallPiToolAsync(string name, JsonElement args, CancellationToken ct);
}

• Step 5: Define ToolRegistry

Create backend/Tools/ToolRegistry.cs:

namespace backend.Tools;

public class ToolRegistry
{
    private readonly Dictionary<string, ITool> _byName;
    private readonly List<ITool> _all;

    public ToolRegistry(IEnumerable<ITool> tools)
    {
        _all = tools.ToList();
        _byName = _all.ToDictionary(t => t.Name);
    }

    public IReadOnlyList<ITool> All => _all;

    public ITool? Get(string name) => _byName.GetValueOrDefault(name);

    public IEnumerable<ITool> EnabledFor(IReadOnlySet<string> enabled)
        => _all.Where(t => enabled.Contains(t.Name));
}

• Step 6: Implement GetCurrentTimeTool

Create backend/Tools/GetCurrentTimeTool.cs:

using System.Text.Json;

namespace backend.Tools;

public class GetCurrentTimeTool : ITool
{
    public string Name => "get_current_time";
    public string Description => "Returns the current UTC time as an ISO-8601 string.";
    public bool RunsDuringResponse => false;

    public JsonElement ParameterSchema { get; } =
        JsonDocument.Parse("""{"type":"object","properties":{},"required":[]}""").RootElement;

    public Task<ToolResult> ExecuteAsync(ToolInvocation invocation, DeviceContext device, CancellationToken ct)
    {
        var now = DateTimeOffset.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ");
        var output = JsonDocument.Parse($$"""{"now":"{{now}}"}""").RootElement;
        return Task.FromResult(new ToolResult(output));
    }
}

• Step 7: Run the tests

Run: dotnet test --filter "ToolRegistryTests|GetCurrentTimeToolTests" Expected: Passed: 3.

• Step 8: Commit

git add backend/Tools/ITool.cs backend/Tools/IDeviceChannel.cs backend/Tools/ToolRegistry.cs backend/Tools/GetCurrentTimeTool.cs backend.tests/ToolRegistryTests.cs backend.tests/GetCurrentTimeToolTests.cs
git commit -m "Add ITool + ToolRegistry + GetCurrentTimeTool"

---

Task 4: EndSessionTool

Files:

• Create: backend/Tools/EndSessionTool.cs
• Create: backend.tests/EndSessionToolTests.cs

end_session has zero parameters and zero side effects. Its return value is {"ok":true} for cleanliness; the actual "close" behaviour is wired in RealtimeSession (Task 12), which special-cases this tool by name. The tool itself is simple: return ok. We mark RunsDuringResponse = false because the relay must wait for response.done before acting on it.

• Step 1: Write the failing test

Create backend.tests/EndSessionToolTests.cs:

using System.Text.Json;
using backend.Tools;
using FluentAssertions;
using Xunit;

namespace backend.tests;

public class EndSessionToolTests
{
    [Fact]
    public async Task Returns_ok_true()
    {
        var tool = new EndSessionTool();
        var ctx = new DeviceContext(Guid.NewGuid(), Guid.NewGuid(), new NoopChannel());

        var res = await tool.ExecuteAsync(
            new ToolInvocation("call_1", JsonDocument.Parse("{}").RootElement), ctx, default);

        res.Output.GetProperty("ok").GetBoolean().Should().BeTrue();
    }

    [Fact]
    public void Name_and_runs_during_response_match_spec()
    {
        var tool = new EndSessionTool();
        tool.Name.Should().Be("end_session");
        tool.RunsDuringResponse.Should().BeFalse();
    }

    private class NoopChannel : IDeviceChannel
    {
        public Task<JsonElement> CallPiToolAsync(string n, JsonElement a, CancellationToken ct)
            => throw new NotSupportedException();
    }
}

• Step 2: Run to see it fail

Run: dotnet test --filter EndSessionToolTests Expected: build error (EndSessionTool not defined).

• Step 3: Implement the tool

Create backend/Tools/EndSessionTool.cs:

using System.Text.Json;

namespace backend.Tools;

public class EndSessionTool : ITool
{
    public string Name => "end_session";
    public string Description =>
        "Call this when the user has said goodbye or otherwise indicates the conversation is over.";
    public bool RunsDuringResponse => false;

    public JsonElement ParameterSchema { get; } =
        JsonDocument.Parse("""{"type":"object","properties":{},"required":[]}""").RootElement;

    public Task<ToolResult> ExecuteAsync(ToolInvocation invocation, DeviceContext device, CancellationToken ct)
    {
        var output = JsonDocument.Parse("""{"ok":true}""").RootElement;
        return Task.FromResult(new ToolResult(output));
    }
}

• Step 4: Run the tests

Run: dotnet test --filter EndSessionToolTests Expected: Passed: 2.

• Step 5: Commit

git add backend/Tools/EndSessionTool.cs backend.tests/EndSessionToolTests.cs
git commit -m "Add EndSessionTool (no params, returns ok; relay closes on this tool name)"

---

Task 5: SetVolumeTool

Files:

• Create: backend/Tools/SetVolumeTool.cs
• Create: backend.tests/SetVolumeToolTests.cs

set_volume runs server-side but immediately delegates to the Pi via IDeviceChannel.CallPiToolAsync. The level is validated (0..100); out-of-range returns {"ok":false,"error":"…"} without ever touching the device.

• Step 1: Write the failing test

Create backend.tests/SetVolumeToolTests.cs:

using System.Text.Json;
using backend.Tools;
using FluentAssertions;
using Xunit;

namespace backend.tests;

public class SetVolumeToolTests
{
    [Fact]
    public async Task Forwards_call_to_pi_and_returns_its_result()
    {
        var fake = new RecordingChannel(
            JsonDocument.Parse("""{"ok":true}""").RootElement);
        var tool = new SetVolumeTool();
        var ctx = new DeviceContext(Guid.NewGuid(), Guid.NewGuid(), fake);

        var args = JsonDocument.Parse("""{"level":42}""").RootElement;
        var res = await tool.ExecuteAsync(new ToolInvocation("c1", args), ctx, default);

        fake.LastName.Should().Be("set_volume");
        fake.LastArgs.GetProperty("level").GetInt32().Should().Be(42);
        res.Output.GetProperty("ok").GetBoolean().Should().BeTrue();
    }

    [Fact]
    public async Task Rejects_out_of_range_level_without_calling_pi()
    {
        var fake = new RecordingChannel(JsonDocument.Parse("""{"ok":true}""").RootElement);
        var tool = new SetVolumeTool();
        var ctx = new DeviceContext(Guid.NewGuid(), Guid.NewGuid(), fake);

        var args = JsonDocument.Parse("""{"level":150}""").RootElement;
        var res = await tool.ExecuteAsync(new ToolInvocation("c1", args), ctx, default);

        fake.LastName.Should().BeNull();
        res.Output.GetProperty("ok").GetBoolean().Should().BeFalse();
        res.Output.GetProperty("error").GetString().Should().Contain("0..100");
    }

    private class RecordingChannel(JsonElement reply) : IDeviceChannel
    {
        public string? LastName { get; private set; }
        public JsonElement LastArgs { get; private set; }
        public Task<JsonElement> CallPiToolAsync(string name, JsonElement args, CancellationToken ct)
        {
            LastName = name;
            LastArgs = args;
            return Task.FromResult(reply);
        }
    }
}

• Step 2: Run to see it fail

Run: dotnet test --filter SetVolumeToolTests Expected: build error (SetVolumeTool not defined).

• Step 3: Implement the tool

Create backend/Tools/SetVolumeTool.cs:

using System.Text.Json;

namespace backend.Tools;

public class SetVolumeTool : ITool
{
    public string Name => "set_volume";
    public string Description =>
        "Sets the assistant's audio output volume on the device. Level is 0..100.";
    public bool RunsDuringResponse => true;

    public JsonElement ParameterSchema { get; } = JsonDocument.Parse(
        """
        {
          "type":"object",
          "properties":{"level":{"type":"integer","minimum":0,"maximum":100}},
          "required":["level"]
        }
        """).RootElement;

    public async Task<ToolResult> ExecuteAsync(ToolInvocation invocation, DeviceContext device, CancellationToken ct)
    {
        if (!invocation.Arguments.TryGetProperty("level", out var lvlEl)
            || !lvlEl.TryGetInt32(out var level)
            || level < 0 || level > 100)
        {
            var err = JsonDocument.Parse("""{"ok":false,"error":"level must be an integer in 0..100"}""").RootElement;
            return new ToolResult(err);
        }

        var piReply = await device.Channel.CallPiToolAsync(Name, invocation.Arguments, ct);
        return new ToolResult(piReply);
    }
}

• Step 4: Run the tests

Run: dotnet test --filter SetVolumeToolTests Expected: Passed: 2.

• Step 5: Commit

git add backend/Tools/SetVolumeTool.cs backend.tests/SetVolumeToolTests.cs
git commit -m "Add SetVolumeTool (validates 0..100, delegates to Pi via IDeviceChannel)"

---

Task 6: OpenAIKeyProvider + RealtimeSettings

Files:

• Create: backend/Realtime/OpenAIKeyProvider.cs
• Create: backend/Realtime/RealtimeSettings.cs

The provider exposes the OpenAI API key sourced from OPENAI_API_KEY env (set as a Coolify secret per spec). RealtimeSettings is the per-conversation snapshot of DeviceConfig that drives session.update.

• Step 1: Implement OpenAIKeyProvider

Create backend/Realtime/OpenAIKeyProvider.cs:

namespace backend.Realtime;

public class OpenAIKeyProvider
{
    private readonly string? _key;

    public OpenAIKeyProvider(IConfiguration cfg)
    {
        _key = cfg["OPENAI_API_KEY"] ?? cfg["OpenAI:ApiKey"];
    }

    public bool HasKey => !string.IsNullOrWhiteSpace(_key);

    public string GetRequired()
    {
        if (string.IsNullOrWhiteSpace(_key))
            throw new InvalidOperationException(
                "OPENAI_API_KEY is not configured. Set the env var on the Coolify app.");
        return _key!;
    }
}

• Step 2: Implement RealtimeSettings

Create backend/Realtime/RealtimeSettings.cs:

namespace backend.Realtime;

public record RealtimeSettings(
    string Model,
    string Voice,
    string SystemPrompt,
    int IdleTimeoutSeconds,
    IReadOnlySet<string> EnabledTools);

• Step 3: Register the provider in DI

In backend/Program.cs, add to the existing service-registration block:

builder.Services.AddSingleton<backend.Realtime.OpenAIKeyProvider>();

• Step 4: Verify build + tests still pass

Run: dotnet test Expected: every prior test passes; nothing new broken.

• Step 5: Commit

git add backend/Realtime/OpenAIKeyProvider.cs backend/Realtime/RealtimeSettings.cs backend/Program.cs
git commit -m "Add OpenAIKeyProvider + RealtimeSettings record"

---

Task 7: IRealtimeUpstream interface + envelope helpers

Files:

• Create: backend/Realtime/RealtimeEvents.cs
• Create: backend/Realtime/IRealtimeUpstream.cs

The relay only sends a tiny handful of upstream event shapes (session.update, input_audio_buffer.append, conversation.item.create with function_call_output, response.create) and reads about a dozen back. Define those as plain JsonNode helpers, not strict types — OpenAI evolves the schema and we never need to model the whole event.

• Step 1: Define event helpers

Create backend/Realtime/RealtimeEvents.cs:

using System.Text.Json;
using System.Text.Json.Nodes;
using backend.Tools;

namespace backend.Realtime;

public static class RealtimeEvents
{
    public static JsonObject SessionUpdate(RealtimeSettings cfg, IEnumerable<ITool> enabledTools)
    {
        var tools = new JsonArray();
        foreach (var t in enabledTools)
        {
            tools.Add(new JsonObject
            {
                ["type"] = "function",
                ["name"] = t.Name,
                ["description"] = t.Description,
                ["parameters"] = JsonNode.Parse(t.ParameterSchema.GetRawText())!,
            });
        }

        return new JsonObject
        {
            ["type"] = "session.update",
            ["session"] = new JsonObject
            {
                ["voice"] = cfg.Voice,
                ["instructions"] = cfg.SystemPrompt,
                ["modalities"] = new JsonArray("text", "audio"),
                ["input_audio_format"] = "pcm16",
                ["output_audio_format"] = "pcm16",
                ["input_audio_transcription"] = new JsonObject { ["model"] = "whisper-1" },
                ["turn_detection"] = new JsonObject
                {
                    ["type"] = "server_vad",
                    ["threshold"] = 0.5,
                    ["prefix_padding_ms"] = 300,
                    ["silence_duration_ms"] = 500,
                },
                ["tools"] = tools,
                ["tool_choice"] = "auto",
            },
        };
    }

    public static JsonObject InputAudioBufferAppend(ReadOnlySpan<byte> pcm16)
    {
        return new JsonObject
        {
            ["type"] = "input_audio_buffer.append",
            ["audio"] = Convert.ToBase64String(pcm16),
        };
    }

    public static JsonObject FunctionCallOutput(string callId, string outputJson)
    {
        return new JsonObject
        {
            ["type"] = "conversation.item.create",
            ["item"] = new JsonObject
            {
                ["type"] = "function_call_output",
                ["call_id"] = callId,
                ["output"] = outputJson,
            },
        };
    }

    public static JsonObject ResponseCreate() => new() { ["type"] = "response.create" };
}

• Step 2: Define IRealtimeUpstream

Create backend/Realtime/IRealtimeUpstream.cs:

using System.Text.Json.Nodes;

namespace backend.Realtime;

public interface IRealtimeUpstream : IAsyncDisposable
{
    Task SendJsonAsync(JsonObject envelope, CancellationToken ct);
    Task<JsonObject?> ReceiveJsonAsync(CancellationToken ct); // null on graceful close
}

• Step 3: Verify build still passes

Run: dotnet build Expected: 0 errors.

• Step 4: Commit

git add backend/Realtime/RealtimeEvents.cs backend/Realtime/IRealtimeUpstream.cs
git commit -m "Add IRealtimeUpstream + RealtimeEvents helpers for upstream JSON"

---

Task 8: ScriptedRealtimeUpstream test double

Files:

• Create: backend.tests/ScriptedRealtimeUpstream.cs
• Create: backend.tests/FakeDeviceChannel.cs

ScriptedRealtimeUpstream lets each test queue the upstream events the relay will see and capture what the relay sends. It's a pure in-memory Channel<JsonObject> pair.

• Step 1: Implement the scripted upstream

Create backend.tests/ScriptedRealtimeUpstream.cs:

using System.Text.Json.Nodes;
using System.Threading.Channels;
using backend.Realtime;

namespace backend.tests;

public class ScriptedRealtimeUpstream : IRealtimeUpstream
{
    private readonly Channel<JsonObject> _toRelay =
        Channel.CreateUnbounded<JsonObject>(new UnboundedChannelOptions { SingleReader = true });

    private readonly Channel<JsonObject> _fromRelay =
        Channel.CreateUnbounded<JsonObject>(new UnboundedChannelOptions { SingleWriter = true });

    public IList<JsonObject> Sent { get; } = new List<JsonObject>();

    public Task SendJsonAsync(JsonObject envelope, CancellationToken ct)
    {
        Sent.Add(envelope);
        return _fromRelay.Writer.WriteAsync(envelope, ct).AsTask();
    }

    public async Task<JsonObject?> ReceiveJsonAsync(CancellationToken ct)
    {
        try
        {
            return await _toRelay.Reader.ReadAsync(ct);
        }
        catch (ChannelClosedException)
        {
            return null;
        }
    }

    public void Push(JsonObject evt) => _toRelay.Writer.TryWrite(evt);
    public void CloseUpstream() => _toRelay.Writer.TryComplete();

    public async Task<JsonObject> WaitForSentAsync(string type, CancellationToken ct)
    {
        await foreach (var item in _fromRelay.Reader.ReadAllAsync(ct))
        {
            if ((string?)item["type"] == type) return item;
        }
        throw new TimeoutException($"never saw outbound {type}");
    }

    public ValueTask DisposeAsync()
    {
        _toRelay.Writer.TryComplete();
        _fromRelay.Writer.TryComplete();
        return ValueTask.CompletedTask;
    }
}

• Step 2: Implement the fake device channel

Create backend.tests/FakeDeviceChannel.cs:

using System.Collections.Concurrent;
using System.Text.Json;
using backend.DeviceHub;
using backend.Tools;

namespace backend.tests;

public class FakeDeviceChannel : IDeviceChannel, IDeviceClient
{
    public List<HubEnvelope> SentEnvelopes { get; } = new();
    public List<byte[]> SentBinary { get; } = new();
    public ConcurrentQueue<JsonElement> PiToolReplies { get; } = new();

    public Task SendEnvelopeAsync<T>(T envelope, CancellationToken ct) where T : HubEnvelope
    {
        SentEnvelopes.Add(envelope);
        return Task.CompletedTask;
    }

    public Task SendBinaryAsync(ReadOnlyMemory<byte> bytes, CancellationToken ct)
    {
        SentBinary.Add(bytes.ToArray());
        return Task.CompletedTask;
    }

    public Task<JsonElement> CallPiToolAsync(string name, JsonElement args, CancellationToken ct)
    {
        if (PiToolReplies.TryDequeue(out var reply))
            return Task.FromResult(reply);
        // Default: succeed silently.
        return Task.FromResult(JsonDocument.Parse("""{"ok":true}""").RootElement);
    }
}

Note: IDeviceClient and HubEnvelope are introduced in Tasks 11 and 16 respectively. This file references them by name so it will compile only after those tasks land. To keep ordering tidy, write this file but git commit it together with whichever of these dependencies arrives first. (Subagent-driven execution should already do this automatically by running dotnet build before commit.) If you're executing inline and want the test double to compile right now, leave only the IDeviceChannel interface and PiToolReplies parts; re-add the IDeviceClient surface during Task 16.

• Step 3: Verify the scripted upstream compiles (DeviceChannel will follow later)

Run: dotnet build backend.tests Expected: errors if FakeDeviceChannel.cs is included before Task 11. Acceptable workaround: temporarily comment out the IDeviceClient/HubEnvelope lines until Task 11 lands; the IDeviceChannel portion is enough for Tasks 9–14.

• Step 4: Commit

git add backend.tests/ScriptedRealtimeUpstream.cs backend.tests/FakeDeviceChannel.cs
git commit -m "Add ScriptedRealtimeUpstream + FakeDeviceChannel test doubles"

---

Task 9: RealtimeSession — lifecycle (open → session.update → session_started → close)

Files:

• Create: backend/Realtime/RealtimeSession.cs
• Create: backend.tests/RealtimeSessionLifecycleTests.cs

The session owns one conversation. RunAsync(CancellationToken):

1. Opens upstream (already constructed by the factory in Task 15; here we accept it via the ctor).
2. Sends session.update, awaits session.updated.
3. Inserts the Conversations row via ConversationLog.
4. Emits session_started to the device.
5. Pumps upstream events (later tasks add audio/transcript/tools).
6. On externally requested cancel OR EndReason set internally, closes upstream, calls ConversationLog.EndAsync, emits session_ended.

In this task we set up the skeleton and verify a happy-path lifecycle: opens, immediately receives session.updated from the scripted upstream, emits session_started, then a CancellationToken cancel triggers session_ended(reason=error). Later tasks add tool/idle paths.

To avoid the WS dependency, this task introduces a tiny output sink interface ISessionOutput that RealtimeSession writes to (envelopes + binary). The DeviceHub adapter in Task 16 implements it by writing to the WS; tests implement it directly.

• Step 1: Write the failing test

Create backend.tests/RealtimeSessionLifecycleTests.cs:

using System.Text.Json.Nodes;
using backend.Conversations;
using backend.Data;
using backend.Realtime;
using backend.Tools;
using FluentAssertions;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class RealtimeSessionLifecycleTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    [Fact]
    public async Task Open_emits_session_update_then_session_started_then_session_ended_on_cancel()
    {
        using var scope = _factory.Services.CreateScope();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var registry = new ToolRegistry(new ITool[] { new GetCurrentTimeTool() });

        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings(
            Model: "gpt-4o-realtime-preview",
            Voice: "alloy",
            SystemPrompt: "test",
            IdleTimeoutSeconds: 30,
            EnabledTools: new HashSet<string> { "get_current_time" });

        var session = new RealtimeSession(
            deviceId: Guid.NewGuid(),
            upstream, sink, log, registry, cfg,
            channel: new FakeDeviceChannel(),
            logger: NullLogger.Instance);

        // Push session.updated so the relay can proceed past the gate.
        upstream.Push(new JsonObject { ["type"] = "session.updated" });

        var cts = new CancellationTokenSource();
        var runTask = session.RunAsync(cts.Token);

        // Wait until session_started is emitted to the device.
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        cts.Cancel();
        await runTask;

        upstream.Sent.Select(e => (string?)e["type"]).Should().StartWith("session.update");
        sink.Envelopes.Select(e => (string?)e["type"]).Should().Contain("session_started");
        sink.Envelopes.Select(e => (string?)e["type"]).Should().Contain("session_ended");
    }
}

(The test references a NullLogger and RecordingSink — add them as helpers in the same file or in a small TestHelpers.cs. For brevity, inline:)

At the bottom of the file, add:

internal class NullLogger : Microsoft.Extensions.Logging.ILogger
{
    public static readonly NullLogger Instance = new();
    public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
    public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => false;
    public void Log<TState>(Microsoft.Extensions.Logging.LogLevel l, Microsoft.Extensions.Logging.EventId e,
        TState s, Exception? ex, Func<TState, Exception?, string> f) { }
}

internal class RecordingSink : ISessionOutput
{
    public List<JsonObject> Envelopes { get; } = new();
    public List<byte[]> Binary { get; } = new();
    public Task WriteEnvelopeAsync(JsonObject e, CancellationToken ct)
    {
        lock (Envelopes) Envelopes.Add(e);
        _gate.Release();
        return Task.CompletedTask;
    }
    public Task WriteBinaryAsync(ReadOnlyMemory<byte> b, CancellationToken ct)
    {
        Binary.Add(b.ToArray());
        return Task.CompletedTask;
    }
    private readonly SemaphoreSlim _gate = new(0, int.MaxValue);
    public async Task WaitForAsync(string type, TimeSpan timeout)
    {
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            lock (Envelopes)
            {
                if (Envelopes.Any(e => (string?)e["type"] == type)) return;
            }
            await _gate.WaitAsync(timeout);
        }
        throw new TimeoutException($"never saw {type}");
    }
}

• Step 2: Run to see it fail

Run: dotnet test --filter RealtimeSessionLifecycleTests Expected: build error (RealtimeSession, ISessionOutput not defined).

• Step 3: Implement ISessionOutput + RealtimeSession

Create backend/Realtime/RealtimeSession.cs:

using System.Text.Json.Nodes;
using backend.Conversations;
using backend.Tools;
using Microsoft.Extensions.Logging;

namespace backend.Realtime;

public interface ISessionOutput
{
    Task WriteEnvelopeAsync(JsonObject envelope, CancellationToken ct);
    Task WriteBinaryAsync(ReadOnlyMemory<byte> bytes, CancellationToken ct);
}

public class RealtimeSession
{
    private readonly Guid _deviceId;
    private readonly IRealtimeUpstream _upstream;
    private readonly ISessionOutput _output;
    private readonly ConversationLog _log;
    private readonly ToolRegistry _registry;
    private readonly RealtimeSettings _cfg;
    private readonly IDeviceChannel _channel;
    private readonly ILogger _logger;

    private Conversation? _conversation;
    private EndReason _endReason = EndReason.Unknown;
    private bool _endRequested;

    public RealtimeSession(
        Guid deviceId,
        IRealtimeUpstream upstream,
        ISessionOutput output,
        ConversationLog log,
        ToolRegistry registry,
        RealtimeSettings cfg,
        IDeviceChannel channel,
        ILogger logger)
    {
        _deviceId = deviceId;
        _upstream = upstream;
        _output = output;
        _log = log;
        _registry = registry;
        _cfg = cfg;
        _channel = channel;
        _logger = logger;
    }

    public Guid? ConversationId => _conversation?.Id;

    public async Task RunAsync(CancellationToken ct)
    {
        try
        {
            var enabled = _registry.EnabledFor(_cfg.EnabledTools).ToList();
            await _upstream.SendJsonAsync(RealtimeEvents.SessionUpdate(_cfg, enabled), ct);

            // Wait for session.updated.
            var ack = await WaitForUpstreamTypeAsync("session.updated", ct);
            if (ack is null)
            {
                _endReason = EndReason.Error;
                return;
            }

            _conversation = await _log.StartAsync(_deviceId, ct);
            await _output.WriteEnvelopeAsync(new JsonObject
            {
                ["type"] = "session_started",
                ["conversation_id"] = _conversation.Id.ToString(),
            }, ct);

            await PumpAsync(ct);
        }
        catch (OperationCanceledException)
        {
            if (_endReason == EndReason.Unknown) _endReason = EndReason.Error;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "RealtimeSession crashed");
            _endReason = EndReason.Error;
        }
        finally
        {
            await _upstream.DisposeAsync();
            if (_conversation is not null)
            {
                await _log.EndAsync(_conversation.Id, _endReason, CancellationToken.None);
            }
            var endEnv = new JsonObject
            {
                ["type"] = "session_ended",
                ["reason"] = _endReason.ToString().ToLowerInvariant(),
            };
            await _output.WriteEnvelopeAsync(endEnv, CancellationToken.None);
        }
    }

    private async Task PumpAsync(CancellationToken ct)
    {
        // Filled in by Task 10+. Loop until cancel or _endRequested.
        while (!ct.IsCancellationRequested && !_endRequested)
        {
            var evt = await _upstream.ReceiveJsonAsync(ct);
            if (evt is null) { _endReason = EndReason.Error; return; }
            // Future tasks dispatch on (string?)evt["type"].
        }
    }

    private async Task<JsonObject?> WaitForUpstreamTypeAsync(string type, CancellationToken ct)
    {
        while (true)
        {
            var evt = await _upstream.ReceiveJsonAsync(ct);
            if (evt is null) return null;
            if ((string?)evt["type"] == type) return evt;
        }
    }
}

• Step 4: Run the test

Run: dotnet test --filter RealtimeSessionLifecycleTests Expected: Passed: 1.

• Step 5: Commit

git add backend/Realtime/RealtimeSession.cs backend.tests/RealtimeSessionLifecycleTests.cs
git commit -m "Add RealtimeSession lifecycle (open + session.update + session_started/ended)"

---

Task 10: RealtimeSession — audio relay (uplink + downlink)

Files:

• Modify: backend/Realtime/RealtimeSession.cs
• Create: backend.tests/RealtimeSessionAudioTests.cs

Uplink: device binary frames push into a Channel<byte[]> exposed by RealtimeSession.WriteUplinkAsync(...). A dedicated pump task base64-encodes each frame and sends it as input_audio_buffer.append. Downlink: when response.audio.delta arrives, decode the base64 delta field and forward as binary to ISessionOutput.

• Step 1: Write the failing test

Create backend.tests/RealtimeSessionAudioTests.cs:

using System.Text.Json.Nodes;
using backend.Conversations;
using backend.Realtime;
using backend.Tools;
using FluentAssertions;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class RealtimeSessionAudioTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    [Fact]
    public async Task Uplink_bytes_become_input_audio_buffer_append()
    {
        using var scope = _factory.Services.CreateScope();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var reg = new ToolRegistry(Array.Empty<ITool>());
        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings("m", "v", "p", 30, new HashSet<string>());
        var session = new RealtimeSession(Guid.NewGuid(), upstream, sink, log, reg, cfg,
            new FakeDeviceChannel(), NullLogger.Instance);

        upstream.Push(new JsonObject { ["type"] = "session.updated" });
        var cts = new CancellationTokenSource();
        var run = session.RunAsync(cts.Token);
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        var frame = new byte[3840];
        for (int i = 0; i < frame.Length; i++) frame[i] = (byte)(i % 256);
        await session.WriteUplinkAsync(frame, default);

        // Spin briefly so the pump can flush.
        await Task.Delay(100);

        var appended = upstream.Sent
            .FirstOrDefault(e => (string?)e["type"] == "input_audio_buffer.append");
        appended.Should().NotBeNull();
        var b64 = (string?)appended!["audio"];
        Convert.FromBase64String(b64!).Length.Should().Be(3840);

        cts.Cancel();
        await run;
    }

    [Fact]
    public async Task Audio_delta_from_upstream_becomes_binary_to_device()
    {
        using var scope = _factory.Services.CreateScope();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var reg = new ToolRegistry(Array.Empty<ITool>());
        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings("m", "v", "p", 30, new HashSet<string>());
        var session = new RealtimeSession(Guid.NewGuid(), upstream, sink, log, reg, cfg,
            new FakeDeviceChannel(), NullLogger.Instance);

        upstream.Push(new JsonObject { ["type"] = "session.updated" });
        var cts = new CancellationTokenSource();
        var run = session.RunAsync(cts.Token);
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        var payload = new byte[3840];
        for (int i = 0; i < payload.Length; i++) payload[i] = (byte)((i + 1) % 256);
        upstream.Push(new JsonObject
        {
            ["type"] = "response.audio.delta",
            ["delta"] = Convert.ToBase64String(payload),
        });

        await Task.Delay(100);
        sink.Binary.Should().NotBeEmpty();
        sink.Binary[0].Length.Should().Be(3840);
        sink.Binary[0][0].Should().Be(1);

        cts.Cancel();
        await run;
    }
}

• Step 2: Run to see it fail

Run: dotnet test --filter RealtimeSessionAudioTests Expected: build error (WriteUplinkAsync doesn't exist).

• Step 3: Add uplink channel + audio.delta handling

In backend/Realtime/RealtimeSession.cs, at the top of the class add fields:

private readonly System.Threading.Channels.Channel<byte[]> _uplink =
    System.Threading.Channels.Channel.CreateBounded<byte[]>(
        new System.Threading.Channels.BoundedChannelOptions(64)
        {
            FullMode = System.Threading.Channels.BoundedChannelFullMode.DropOldest,
        });

Add a public uplink writer (used by the device hub):

public ValueTask WriteUplinkAsync(byte[] frame, CancellationToken ct)
    => _uplink.Writer.WriteAsync(frame, ct);

In RunAsync, after emitting session_started, launch the uplink pump alongside PumpAsync:

var uplinkTask = Task.Run(() => PumpUplinkAsync(ct), ct);
await PumpAsync(ct);
_uplink.Writer.TryComplete();
await uplinkTask;

Add the pump method:

private async Task PumpUplinkAsync(CancellationToken ct)
{
    try
    {
        await foreach (var frame in _uplink.Reader.ReadAllAsync(ct))
        {
            await _upstream.SendJsonAsync(RealtimeEvents.InputAudioBufferAppend(frame), ct);
        }
    }
    catch (OperationCanceledException) { }
    catch (Exception ex) { _logger.LogWarning(ex, "uplink pump errored"); }
}

Extend PumpAsync to handle response.audio.delta — replace the existing loop body with:

while (!ct.IsCancellationRequested && !_endRequested)
{
    var evt = await _upstream.ReceiveJsonAsync(ct);
    if (evt is null) { _endReason = EndReason.Error; return; }
    var type = (string?)evt["type"];
    switch (type)
    {
        case "response.audio.delta":
            await ForwardAudioDeltaAsync(evt, ct);
            break;
    }
}

And add the helper:

private async Task ForwardAudioDeltaAsync(JsonObject evt, CancellationToken ct)
{
    var b64 = (string?)evt["delta"];
    if (string.IsNullOrEmpty(b64)) return;
    var bytes = Convert.FromBase64String(b64);
    await _output.WriteBinaryAsync(bytes, ct);
}

• Step 4: Run the tests

Run: dotnet test --filter RealtimeSessionAudioTests Expected: Passed: 2.

• Step 5: Commit

git add backend/Realtime/RealtimeSession.cs backend.tests/RealtimeSessionAudioTests.cs
git commit -m "RealtimeSession: uplink pump + downlink audio.delta forwarding"

---

Task 11: RealtimeSession — transcript persistence + assistant_done

Files:

• Modify: backend/Realtime/RealtimeSession.cs
• Create: backend.tests/RealtimeSessionTranscriptTests.cs

Handle three upstream events:

• response.audio_transcript.done → transcript field, append assistant turn.

• conversation.item.input_audio_transcription.completed → transcript field, append user turn.

• response.done → emit assistant_done to the device.

• Step 1: Write the failing test

Create backend.tests/RealtimeSessionTranscriptTests.cs:

using System.Text.Json.Nodes;
using backend.Conversations;
using backend.Data;
using backend.Realtime;
using backend.Tools;
using FluentAssertions;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class RealtimeSessionTranscriptTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    [Fact]
    public async Task User_and_assistant_transcripts_are_persisted_and_response_done_emits_assistant_done()
    {
        using var scope = _factory.Services.CreateScope();
        var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var reg = new ToolRegistry(Array.Empty<ITool>());
        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings("m", "v", "p", 30, new HashSet<string>());
        var session = new RealtimeSession(Guid.NewGuid(), upstream, sink, log, reg, cfg,
            new FakeDeviceChannel(), NullLogger.Instance);

        upstream.Push(new JsonObject { ["type"] = "session.updated" });
        var cts = new CancellationTokenSource();
        var run = session.RunAsync(cts.Token);
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        upstream.Push(new JsonObject
        {
            ["type"] = "conversation.item.input_audio_transcription.completed",
            ["transcript"] = "hello",
        });
        upstream.Push(new JsonObject
        {
            ["type"] = "response.audio_transcript.done",
            ["transcript"] = "hi there",
        });
        upstream.Push(new JsonObject { ["type"] = "response.done" });

        await sink.WaitForAsync("assistant_done", TimeSpan.FromSeconds(2));

        var convoId = session.ConversationId!.Value;
        var turns = await db.Turns
            .Where(t => t.ConversationId == convoId)
            .OrderBy(t => t.Index).ToListAsync();
        turns.Select(t => t.Role).Should().Equal(TurnRole.User, TurnRole.Assistant);
        turns[0].Text.Should().Be("hello");
        turns[1].Text.Should().Be("hi there");

        cts.Cancel();
        await run;
    }
}

• Step 2: Run to see it fail

Run: dotnet test --filter RealtimeSessionTranscriptTests Expected: 1 failure (events not yet handled).

• Step 3: Extend the dispatch switch

In backend/Realtime/RealtimeSession.cs PumpAsync's switch, add:

case "conversation.item.input_audio_transcription.completed":
{
    var text = (string?)evt["transcript"];
    if (_conversation is not null && !string.IsNullOrEmpty(text))
        await _log.AppendUserAsync(_conversation.Id, text!, ct);
    break;
}
case "response.audio_transcript.done":
{
    var text = (string?)evt["transcript"];
    if (_conversation is not null && !string.IsNullOrEmpty(text))
        await _log.AppendAssistantAsync(_conversation.Id, text!, ct);
    break;
}
case "response.done":
    await _output.WriteEnvelopeAsync(
        new JsonObject { ["type"] = "assistant_done" }, ct);
    break;

• Step 4: Run the test

Run: dotnet test --filter RealtimeSessionTranscriptTests Expected: Passed: 1.

• Step 5: Commit

git add backend/Realtime/RealtimeSession.cs backend.tests/RealtimeSessionTranscriptTests.cs
git commit -m "RealtimeSession: persist user/assistant transcripts; emit assistant_done"

---

Task 12: RealtimeSession — server-side tool calls (get_current_time)

Files:

• Modify: backend/Realtime/RealtimeSession.cs
• Create: backend.tests/RealtimeSessionToolsTests.cs

A tool call surfaces on the upstream as response.function_call_arguments.done with name, call_id, and arguments (a stringified JSON). The relay must:

1. Collect these during the response (do not execute yet — wait for response.done).
2. At response.done:
   • If any collected call has name == "end_session": defer to Task 13 (end-session sequencing).
   • Otherwise for each call: invoke the tool, persist a Turn(role=tool), send conversation.item.create (function_call_output) + response.create upstream.

Track pending calls in a list on the session.

• Step 1: Write the failing test

Create backend.tests/RealtimeSessionToolsTests.cs:

using System.Text.Json.Nodes;
using backend.Conversations;
using backend.Data;
using backend.Realtime;
using backend.Tools;
using FluentAssertions;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class RealtimeSessionToolsTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    [Fact]
    public async Task Get_current_time_call_returns_function_call_output_and_persists_tool_turn()
    {
        using var scope = _factory.Services.CreateScope();
        var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var reg = new ToolRegistry(new ITool[] { new GetCurrentTimeTool() });
        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings("m", "v", "p", 30, new HashSet<string> { "get_current_time" });
        var session = new RealtimeSession(Guid.NewGuid(), upstream, sink, log, reg, cfg,
            new FakeDeviceChannel(), NullLogger.Instance);

        upstream.Push(new JsonObject { ["type"] = "session.updated" });
        var cts = new CancellationTokenSource();
        var run = session.RunAsync(cts.Token);
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        upstream.Push(new JsonObject
        {
            ["type"] = "response.function_call_arguments.done",
            ["call_id"] = "call_abc",
            ["name"] = "get_current_time",
            ["arguments"] = "{}",
        });
        upstream.Push(new JsonObject { ["type"] = "response.done" });

        // Wait for function_call_output + response.create to appear on upstream.
        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(2);
        while (DateTime.UtcNow < deadline && !upstream.Sent.Any(e =>
            (string?)e["type"] == "conversation.item.create"))
        {
            await Task.Delay(20);
        }

        var item = upstream.Sent.First(e => (string?)e["type"] == "conversation.item.create");
        ((string?)item["item"]!["type"]).Should().Be("function_call_output");
        ((string?)item["item"]!["call_id"]).Should().Be("call_abc");
        ((string?)item["item"]!["output"]).Should().Contain("\"now\"");

        upstream.Sent.Should().Contain(e => (string?)e["type"] == "response.create");

        var convoId = session.ConversationId!.Value;
        var toolTurn = await db.Turns.FirstAsync(t =>
            t.ConversationId == convoId && t.Role == TurnRole.Tool);
        toolTurn.ToolName.Should().Be("get_current_time");
        toolTurn.ToolResultJson.Should().Contain("\"now\"");

        cts.Cancel();
        await run;
    }
}

• Step 2: Run to see it fail

Run: dotnet test --filter RealtimeSessionToolsTests Expected: 1 failure.

• Step 3: Track + execute pending tool calls

In backend/Realtime/RealtimeSession.cs, add a field at the top of the class:

private readonly List<(string CallId, string Name, string ArgsJson)> _pendingCalls = new();

In PumpAsync's switch, add:

case "response.function_call_arguments.done":
{
    var callId = (string?)evt["call_id"] ?? "";
    var name = (string?)evt["name"] ?? "";
    var args = (string?)evt["arguments"] ?? "{}";
    _pendingCalls.Add((callId, name, args));
    break;
}

Replace the existing case "response.done" with:

case "response.done":
{
    await _output.WriteEnvelopeAsync(
        new JsonObject { ["type"] = "assistant_done" }, ct);

    if (_pendingCalls.Count == 0) break;
    var calls = _pendingCalls.ToList();
    _pendingCalls.Clear();

    if (calls.Any(c => c.Name == "end_session"))
    {
        // Handled in Task 13.
        _endReason = EndReason.Tool;
        _endRequested = true;
        break;
    }

    foreach (var c in calls)
    {
        await ExecuteToolAsync(c.CallId, c.Name, c.ArgsJson, ct);
    }
    break;
}

Add the helper:

private async Task ExecuteToolAsync(string callId, string name, string argsJson, CancellationToken ct)
{
    var tool = _registry.Get(name);
    string outputJson;
    if (tool is null)
    {
        outputJson = $$"""{"ok":false,"error":"unknown tool: {{name}}"}""";
    }
    else
    {
        try
        {
            using var argsDoc = System.Text.Json.JsonDocument.Parse(argsJson);
            var ctx = new DeviceContext(_deviceId, _conversation!.Id, _channel);
            var res = await tool.ExecuteAsync(
                new ToolInvocation(callId, argsDoc.RootElement.Clone()), ctx, ct);
            outputJson = res.Output.GetRawText();
        }
        catch (Exception ex)
        {
            outputJson = System.Text.Json.JsonSerializer.Serialize(new
            {
                ok = false,
                error = ex.Message,
            });
        }
    }

    if (_conversation is not null)
    {
        await _log.AppendToolAsync(_conversation.Id, name, argsJson, outputJson, ct);
    }

    await _upstream.SendJsonAsync(RealtimeEvents.FunctionCallOutput(callId, outputJson), ct);
    await _upstream.SendJsonAsync(RealtimeEvents.ResponseCreate(), ct);
}

• Step 4: Run the test

Run: dotnet test --filter RealtimeSessionToolsTests Expected: Passed: 1.

• Step 5: Commit

git add backend/Realtime/RealtimeSession.cs backend.tests/RealtimeSessionToolsTests.cs
git commit -m "RealtimeSession: execute server-side tools at response.done"

---

Task 13: RealtimeSession — end_session sequencing

Files:

• Modify: backend/Realtime/RealtimeSession.cs
• Create: backend.tests/RealtimeSessionEndSessionTests.cs

When end_session is one of the pending calls at response.done, the relay:

1. Sets _endReason = EndReason.Tool and _endRequested = true.
2. Does NOT send a response.create.
3. Closes the upstream and emits session_ended(reason=tool) (this is already handled by RunAsync's finally).

Already wired in Task 12 — this task adds the test that proves it, and tightens the audio-draining behaviour so the assistant audio that arrived BEFORE response.done is forwarded (also already true, since audio.delta is forwarded eagerly).

• Step 1: Write the failing test

Create backend.tests/RealtimeSessionEndSessionTests.cs:

using System.Text.Json.Nodes;
using backend.Conversations;
using backend.Realtime;
using backend.Tools;
using FluentAssertions;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class RealtimeSessionEndSessionTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    [Fact]
    public async Task End_session_call_closes_session_with_reason_tool_and_skips_response_create()
    {
        using var scope = _factory.Services.CreateScope();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var reg = new ToolRegistry(new ITool[] { new EndSessionTool() });
        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings("m", "v", "p", 30, new HashSet<string> { "end_session" });
        var session = new RealtimeSession(Guid.NewGuid(), upstream, sink, log, reg, cfg,
            new FakeDeviceChannel(), NullLogger.Instance);

        upstream.Push(new JsonObject { ["type"] = "session.updated" });
        var run = session.RunAsync(default);
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        // Push some audio so we can confirm it is forwarded before close.
        upstream.Push(new JsonObject
        {
            ["type"] = "response.audio.delta",
            ["delta"] = Convert.ToBase64String(new byte[3840]),
        });
        upstream.Push(new JsonObject
        {
            ["type"] = "response.function_call_arguments.done",
            ["call_id"] = "c1",
            ["name"] = "end_session",
            ["arguments"] = "{}",
        });
        upstream.Push(new JsonObject { ["type"] = "response.done" });

        await run;

        // Audio forwarded before close.
        sink.Binary.Should().NotBeEmpty();

        // session_ended must have reason=tool.
        var ended = sink.Envelopes.Last(e => (string?)e["type"] == "session_ended");
        ((string?)ended["reason"]).Should().Be("tool");

        // No response.create was sent upstream.
        upstream.Sent.Should().NotContain(e => (string?)e["type"] == "response.create");
    }
}

• Step 2: Run the test

Run: dotnet test --filter RealtimeSessionEndSessionTests Expected: Passed: 1 (logic already in place from Task 12; if not, the test will pinpoint the gap to fix).

• Step 3: Commit

git add backend.tests/RealtimeSessionEndSessionTests.cs
git commit -m "Test end_session sequencing closes with reason=tool and skips response.create"

---

Task 14: RealtimeSession — idle timeout

Files:

• Modify: backend/Realtime/RealtimeSession.cs
• Create: backend.tests/RealtimeSessionIdleTimeoutTests.cs

If IdleTimeoutSeconds elapses with no input_audio_buffer.speech_started upstream event, end the session with reason=idle. Each speech_started resets the timer.

Implementation: a DateTime field _lastSpeechAt, updated on speech_started. A background task in RunAsync checks every 500 ms; if DateTime.UtcNow - _lastSpeechAt > timeout AND no response.done is pending (i.e. we're not mid-assistant-speech), set _endReason = Idle and _endRequested = true.

For the test, we'll use a 1-second timeout to keep it fast.

• Step 1: Write the failing test

Create backend.tests/RealtimeSessionIdleTimeoutTests.cs:

using System.Text.Json.Nodes;
using backend.Conversations;
using backend.Realtime;
using backend.Tools;
using FluentAssertions;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class RealtimeSessionIdleTimeoutTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    [Fact]
    public async Task Session_ends_with_reason_idle_when_no_speech_for_timeout()
    {
        using var scope = _factory.Services.CreateScope();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var reg = new ToolRegistry(Array.Empty<ITool>());
        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings("m", "v", "p", IdleTimeoutSeconds: 1, new HashSet<string>());
        var session = new RealtimeSession(Guid.NewGuid(), upstream, sink, log, reg, cfg,
            new FakeDeviceChannel(), NullLogger.Instance);

        upstream.Push(new JsonObject { ["type"] = "session.updated" });
        var run = session.RunAsync(default);
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        // Wait > 1.5 s without sending any speech_started.
        await run.WaitAsync(TimeSpan.FromSeconds(4));

        var ended = sink.Envelopes.Last(e => (string?)e["type"] == "session_ended");
        ((string?)ended["reason"]).Should().Be("idle");
    }

    [Fact]
    public async Task Speech_started_resets_the_idle_timer()
    {
        using var scope = _factory.Services.CreateScope();
        var log = scope.ServiceProvider.GetRequiredService<ConversationLog>();
        var reg = new ToolRegistry(Array.Empty<ITool>());
        var upstream = new ScriptedRealtimeUpstream();
        var sink = new RecordingSink();
        var cfg = new RealtimeSettings("m", "v", "p", IdleTimeoutSeconds: 2, new HashSet<string>());
        var session = new RealtimeSession(Guid.NewGuid(), upstream, sink, log, reg, cfg,
            new FakeDeviceChannel(), NullLogger.Instance);

        upstream.Push(new JsonObject { ["type"] = "session.updated" });
        var run = session.RunAsync(default);
        await sink.WaitForAsync("session_started", TimeSpan.FromSeconds(2));

        // Wait 1 s, ping speech, wait 1.5 s — total 2.5 s, but timer reset at 1 s, so still alive at 2.5 s.
        await Task.Delay(1000);
        upstream.Push(new JsonObject { ["type"] = "input_audio_buffer.speech_started" });
        await Task.Delay(1500);

        sink.Envelopes.Any(e => (string?)e["type"] == "session_ended").Should().BeFalse();

        // Then let it actually idle out.
        await run.WaitAsync(TimeSpan.FromSeconds(4));
        var ended = sink.Envelopes.Last(e => (string?)e["type"] == "session_ended");
        ((string?)ended["reason"]).Should().Be("idle");
    }
}

• Step 2: Run to see them fail

Run: dotnet test --filter RealtimeSessionIdleTimeoutTests Expected: 2 failures (no idle-timeout logic yet).

• Step 3: Add the timer

In backend/Realtime/RealtimeSession.cs, add a field:

private DateTime _lastSpeechAt = DateTime.UtcNow;

In PumpAsync's switch, add:

case "input_audio_buffer.speech_started":
    _lastSpeechAt = DateTime.UtcNow;
    break;

Reset _lastSpeechAt = DateTime.UtcNow; immediately after emitting session_started in RunAsync (so the first turn has a fresh window).

In RunAsync, after var uplinkTask = Task.Run(...), start the idle watchdog:

var idleTask = Task.Run(() => WatchdogAsync(ct), ct);

And after await PumpAsync(ct);, before _uplink.Writer.TryComplete();, add:

await idleTask;

Add the method:

private async Task WatchdogAsync(CancellationToken ct)
{
    try
    {
        var timeout = TimeSpan.FromSeconds(_cfg.IdleTimeoutSeconds);
        while (!ct.IsCancellationRequested && !_endRequested)
        {
            await Task.Delay(250, ct);
            if (DateTime.UtcNow - _lastSpeechAt > timeout)
            {
                _endReason = EndReason.Idle;
                _endRequested = true;
                return;
            }
        }
    }
    catch (OperationCanceledException) { }
}

Because PumpAsync blocks in ReceiveJsonAsync, the watchdog setting _endRequested = true won't immediately exit the loop. Tighten PumpAsync so it polls upstream with a small linger:

Replace await _upstream.ReceiveJsonAsync(ct) with a wrapper that yields if the upstream has no traffic for 250 ms:

var evt = await ReceiveOrIdleAsync(ct);
if (evt is null && _endRequested) return;
if (evt is null && ct.IsCancellationRequested) return;
if (evt is null) continue;

Add the helper:

private async Task<JsonObject?> ReceiveOrIdleAsync(CancellationToken ct)
{
    using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
    var receive = _upstream.ReceiveJsonAsync(timeout.Token).AsTaskOrFromResult();
    var delay = Task.Delay(250, timeout.Token);
    var done = await Task.WhenAny(receive, delay);
    if (done == receive) return await receive;
    timeout.Cancel(); // unhook the receive task; relay tolerates a re-poll
    return null;
}

IRealtimeUpstream.ReceiveJsonAsync returns Task<JsonObject?> so AsTaskOrFromResult() is a tiny adapter — drop it; just use _upstream.ReceiveJsonAsync(timeout.Token). Update the helper to:

private async Task<JsonObject?> ReceiveOrIdleAsync(CancellationToken ct)
{
    using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
    var receive = _upstream.ReceiveJsonAsync(timeout.Token);
    var delay = Task.Delay(250, timeout.Token);
    var done = await Task.WhenAny(receive, delay);
    if (done == receive) return await receive;
    timeout.Cancel();
    try { await receive; } catch { /* ignore cancellation */ }
    return null;
}

(ScriptedRealtimeUpstream.ReceiveJsonAsync already throws / returns null on cancel, so the discarded await is safe.)

• Step 4: Run the tests

Run: dotnet test --filter RealtimeSessionIdleTimeoutTests Expected: Passed: 2.

• Step 5: Commit

git add backend/Realtime/RealtimeSession.cs backend.tests/RealtimeSessionIdleTimeoutTests.cs
git commit -m "RealtimeSession: idle-timeout watchdog (resets on speech_started)"

---

Task 15: OpenAIRealtimeUpstream (real ClientWebSocket implementation) + RealtimeSessionFactory

Files:

• Create: backend/Realtime/OpenAIRealtimeUpstream.cs
• Create: backend/Realtime/RealtimeSessionFactory.cs
• Modify: backend/Program.cs

No unit tests — exercised only by the live smoke (Task 18). Manual confidence: the upstream is small (open WSS with two headers, send/receive JSON text frames) and reading is a tight ReceiveAsync loop reassembling fragmented messages.

• Step 1: Implement the upstream

Create backend/Realtime/OpenAIRealtimeUpstream.cs:

using System.Net.WebSockets;
using System.Text;
using System.Text.Json.Nodes;

namespace backend.Realtime;

public class OpenAIRealtimeUpstream : IRealtimeUpstream
{
    private readonly ClientWebSocket _ws = new();

    public async Task ConnectAsync(string apiKey, string model, CancellationToken ct)
    {
        _ws.Options.SetRequestHeader("Authorization", $"Bearer {apiKey}");
        _ws.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");
        var uri = new Uri($"wss://api.openai.com/v1/realtime?model={Uri.EscapeDataString(model)}");
        await _ws.ConnectAsync(uri, ct);
    }

    public async Task SendJsonAsync(JsonObject envelope, CancellationToken ct)
    {
        var json = envelope.ToJsonString();
        var bytes = Encoding.UTF8.GetBytes(json);
        await _ws.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, ct);
    }

    public async Task<JsonObject?> ReceiveJsonAsync(CancellationToken ct)
    {
        var buffer = new byte[8192];
        using var stream = new MemoryStream();
        while (true)
        {
            WebSocketReceiveResult result;
            try { result = await _ws.ReceiveAsync(buffer, ct); }
            catch (WebSocketException) { return null; }
            catch (OperationCanceledException) { return null; }

            if (result.MessageType == WebSocketMessageType.Close) return null;
            stream.Write(buffer, 0, result.Count);
            if (result.EndOfMessage)
            {
                var json = Encoding.UTF8.GetString(stream.ToArray());
                return JsonNode.Parse(json)?.AsObject();
            }
        }
    }

    public async ValueTask DisposeAsync()
    {
        if (_ws.State == WebSocketState.Open)
        {
            try { await _ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "bye", CancellationToken.None); }
            catch { }
        }
        _ws.Dispose();
    }
}

• Step 2: Implement RealtimeSessionFactory

Create backend/Realtime/RealtimeSessionFactory.cs:

using backend.Conversations;
using backend.Tools;
using Microsoft.Extensions.Logging;

namespace backend.Realtime;

public class RealtimeSessionFactory(
    OpenAIKeyProvider keys,
    ToolRegistry registry,
    ConversationLog log,
    ILoggerFactory loggerFactory)
{
    public async Task<RealtimeSession> CreateAsync(
        Guid deviceId,
        RealtimeSettings cfg,
        ISessionOutput output,
        IDeviceChannel channel,
        CancellationToken ct)
    {
        var upstream = new OpenAIRealtimeUpstream();
        await upstream.ConnectAsync(keys.GetRequired(), cfg.Model, ct);
        return new RealtimeSession(
            deviceId, upstream, output, log, registry, cfg, channel,
            loggerFactory.CreateLogger<RealtimeSession>());
    }
}

• Step 3: Register in DI

In backend/Program.cs, add:

builder.Services.AddSingleton<backend.Tools.ITool, backend.Tools.GetCurrentTimeTool>();
builder.Services.AddSingleton<backend.Tools.ITool, backend.Tools.EndSessionTool>();
builder.Services.AddSingleton<backend.Tools.ITool, backend.Tools.SetVolumeTool>();
builder.Services.AddSingleton<backend.Tools.ToolRegistry>();
builder.Services.AddScoped<backend.Realtime.RealtimeSessionFactory>();

• Step 4: Build verify

Run: dotnet build Expected: 0 errors. The previous tests continue to pass (no behaviour change to RealtimeSession).

• Step 5: Commit

git add backend/Realtime/OpenAIRealtimeUpstream.cs backend/Realtime/RealtimeSessionFactory.cs backend/Program.cs
git commit -m "Add OpenAIRealtimeUpstream + RealtimeSessionFactory + DI registrations"

---

Task 16: ActiveDevice + DeviceRegistry + HubEnvelopes + IDeviceClient

Files:

• Create: backend/DeviceHub/HubEnvelopes.cs
• Create: backend/DeviceHub/ActiveDevice.cs
• Create: backend/DeviceHub/DeviceRegistry.cs
• Create: backend.tests/DeviceRegistryTests.cs
• Modify: backend.tests/FakeDeviceChannel.cs (uncomment the IDeviceClient/HubEnvelope references introduced in Task 8)

Envelopes are records with [JsonPropertyName] for the snake-case fields used on the wire. ActiveDevice owns one WS, implements IDeviceChannel (for tools) and IDeviceClient (for the relay → device output). It also manages tool_result correlation via a ConcurrentDictionary<callId, TaskCompletionSource<JsonElement>>.

• Step 1: Define envelopes

Create backend/DeviceHub/HubEnvelopes.cs:

using System.Text.Json;
using System.Text.Json.Serialization;

namespace backend.DeviceHub;

public record HubEnvelope([property: JsonPropertyName("type")] string Type);

public record HelloEnvelope(string Type,
    [property: JsonPropertyName("device_id")] Guid DeviceId,
    [property: JsonPropertyName("client_version")] string ClientVersion) : HubEnvelope(Type);

public record HelloAckEnvelope(string Type,
    [property: JsonPropertyName("config")] JsonElement Config) : HubEnvelope(Type);

public record PongEnvelope(string Type) : HubEnvelope(Type);

public record WakeEnvelope(string Type,
    [property: JsonPropertyName("at")] long? At) : HubEnvelope(Type);

public record SessionStartedEnvelope(string Type,
    [property: JsonPropertyName("conversation_id")] Guid ConversationId) : HubEnvelope(Type);

public record SessionEndedEnvelope(string Type,
    [property: JsonPropertyName("reason")] string Reason) : HubEnvelope(Type);

public record AssistantDoneEnvelope(string Type) : HubEnvelope(Type);

public record ToolCallEnvelope(string Type,
    [property: JsonPropertyName("call_id")] string CallId,
    [property: JsonPropertyName("name")] string Name,
    [property: JsonPropertyName("arguments")] JsonElement Arguments) : HubEnvelope(Type);

public record ToolResultEnvelope(string Type,
    [property: JsonPropertyName("call_id")] string CallId,
    [property: JsonPropertyName("ok")] bool Ok,
    [property: JsonPropertyName("result")] JsonElement? Result,
    [property: JsonPropertyName("error")] string? Error) : HubEnvelope(Type);

public record ErrorEnvelope(string Type,
    [property: JsonPropertyName("code")] string Code,
    [property: JsonPropertyName("message")] string Message,
    [property: JsonPropertyName("fatal")] bool? Fatal) : HubEnvelope(Type);

• Step 2: Define IDeviceClient

In a new section of backend/DeviceHub/ActiveDevice.cs (file written next step), the relay calls these to push to the device. Declare interface inline at the top:

public interface IDeviceClient
{
    Task SendEnvelopeAsync<T>(T envelope, CancellationToken ct) where T : HubEnvelope;
    Task SendBinaryAsync(ReadOnlyMemory<byte> bytes, CancellationToken ct);
}

• Step 3: Implement ActiveDevice

Create backend/DeviceHub/ActiveDevice.cs:

using System.Collections.Concurrent;
using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
using backend.Realtime;
using backend.Tools;

namespace backend.DeviceHub;

public interface IDeviceClient
{
    Task SendEnvelopeAsync<T>(T envelope, CancellationToken ct) where T : HubEnvelope;
    Task SendBinaryAsync(ReadOnlyMemory<byte> bytes, CancellationToken ct);
}

public class ActiveDevice : IDeviceChannel, IDeviceClient, ISessionOutput, IAsyncDisposable
{
    public Guid DeviceId { get; }
    private readonly WebSocket _ws;
    private readonly SemaphoreSlim _sendLock = new(1, 1);
    private readonly ConcurrentDictionary<string, TaskCompletionSource<JsonElement>> _pendingTools = new();

    public ActiveDevice(Guid deviceId, WebSocket ws)
    {
        DeviceId = deviceId;
        _ws = ws;
    }

    public async Task SendEnvelopeAsync<T>(T envelope, CancellationToken ct) where T : HubEnvelope
    {
        var json = JsonSerializer.Serialize(envelope, envelope.GetType());
        await _sendLock.WaitAsync(ct);
        try
        {
            await _ws.SendAsync(Encoding.UTF8.GetBytes(json),
                WebSocketMessageType.Text, endOfMessage: true, ct);
        }
        finally { _sendLock.Release(); }
    }

    public async Task SendBinaryAsync(ReadOnlyMemory<byte> bytes, CancellationToken ct)
    {
        await _sendLock.WaitAsync(ct);
        try
        {
            await _ws.SendAsync(bytes, WebSocketMessageType.Binary, endOfMessage: true, ct);
        }
        finally { _sendLock.Release(); }
    }

    public Task WriteEnvelopeAsync(JsonObject envelope, CancellationToken ct)
    {
        // Relay uses JsonObject for upstream; we marshal back to a string here.
        var json = envelope.ToJsonString();
        return WriteRawTextAsync(json, ct);
    }

    private async Task WriteRawTextAsync(string json, CancellationToken ct)
    {
        await _sendLock.WaitAsync(ct);
        try
        {
            await _ws.SendAsync(Encoding.UTF8.GetBytes(json),
                WebSocketMessageType.Text, endOfMessage: true, ct);
        }
        finally { _sendLock.Release(); }
    }

    public Task WriteBinaryAsync(ReadOnlyMemory<byte> bytes, CancellationToken ct)
        => SendBinaryAsync(bytes, ct);

    public Task<JsonElement> CallPiToolAsync(string name, JsonElement args, CancellationToken ct)
    {
        var callId = "srv_" + Guid.NewGuid().ToString("N")[..12];
        var tcs = new TaskCompletionSource<JsonElement>(TaskCreationOptions.RunContinuationsAsynchronously);
        _pendingTools[callId] = tcs;

        var envelope = new ToolCallEnvelope("tool_call", callId, name, args);
        _ = SendEnvelopeAsync(envelope, ct);

        ct.Register(() => tcs.TrySetCanceled());
        return tcs.Task;
    }

    public void CompleteToolResult(ToolResultEnvelope env)
    {
        if (!_pendingTools.TryRemove(env.CallId, out var tcs)) return;
        if (env.Ok && env.Result is JsonElement r) tcs.TrySetResult(r);
        else
        {
            var doc = JsonDocument.Parse($$"""{"ok":false,"error":"{{env.Error ?? "unknown"}}"}""");
            tcs.TrySetResult(doc.RootElement.Clone());
        }
    }

    public async ValueTask DisposeAsync()
    {
        try
        {
            if (_ws.State == WebSocketState.Open)
                await _ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "bye", CancellationToken.None);
        }
        catch { }
        _ws.Dispose();
        _sendLock.Dispose();
    }
}

• Step 4: Implement DeviceRegistry

Create backend/DeviceHub/DeviceRegistry.cs:

using System.Collections.Concurrent;

namespace backend.DeviceHub;

public class DeviceRegistry
{
    private readonly ConcurrentDictionary<Guid, ActiveDevice> _online = new();

    public bool TryRegister(ActiveDevice dev)
        => _online.TryAdd(dev.DeviceId, dev);

    public bool Unregister(Guid deviceId)
        => _online.TryRemove(deviceId, out _);

    public ActiveDevice? Get(Guid deviceId)
        => _online.TryGetValue(deviceId, out var d) ? d : null;

    public bool IsOnline(Guid deviceId) => _online.ContainsKey(deviceId);

    public int OnlineCount => _online.Count;
}

• Step 5: Write the registry test

Create backend.tests/DeviceRegistryTests.cs:

using System.Net.WebSockets;
using backend.DeviceHub;
using FluentAssertions;
using Xunit;

namespace backend.tests;

public class DeviceRegistryTests
{
    [Fact]
    public void Register_unregister_round_trip()
    {
        var reg = new DeviceRegistry();
        var id = Guid.NewGuid();
        var dev = new ActiveDevice(id, new ClientWebSocket()); // not connected — never sends
        reg.TryRegister(dev).Should().BeTrue();
        reg.TryRegister(dev).Should().BeFalse();
        reg.IsOnline(id).Should().BeTrue();
        reg.OnlineCount.Should().Be(1);

        reg.Unregister(id).Should().BeTrue();
        reg.IsOnline(id).Should().BeFalse();
        reg.OnlineCount.Should().Be(0);
    }
}

• Step 6: Update FakeDeviceChannel (if you stubbed it in Task 8)

If you stubbed out the IDeviceClient/HubEnvelope lines back in Task 8, uncomment / restore them now. The full file (Task 8 step 2) is the final version.

• Step 7: Register in DI

In backend/Program.cs, add:

builder.Services.AddSingleton<backend.DeviceHub.DeviceRegistry>();

• Step 8: Run tests

Run: dotnet test --filter DeviceRegistryTests Expected: Passed: 1. The full suite still passes.

• Step 9: Commit

git add backend/DeviceHub backend.tests/DeviceRegistryTests.cs backend.tests/FakeDeviceChannel.cs backend/Program.cs
git commit -m "Add ActiveDevice + DeviceRegistry + HubEnvelopes + IDeviceClient/Channel implementations"

---

Task 17: DeviceHubEndpoint — bearer auth + WS upgrade + handshake + ping/pong

Files:

• Create: backend/DeviceHub/DeviceAuth.cs
• Create: backend/DeviceHub/DeviceHubEndpoint.cs
• Create: backend.tests/DeviceHubHandshakeTests.cs
• Modify: backend/Program.cs

The endpoint:

1. Reads Authorization: Bearer <token> from the upgrade request.
2. Looks up the device by TokenHash (DeviceTokenService.Hash + DB query).
3. Rejects with HTTP 401 if not found or revoked.
4. Upgrades the WS.
5. Reads the first text frame; expects hello; responds with hello_ack carrying the snapshot config.
6. Loops: dispatch by type. Implements ping → updates LastSeenAt, replies pong.
7. Cleanly closes on cancel / client disconnect; removes from DeviceRegistry.

This task ships handshake + ping/pong only. Wake/audio routing is Task 18.

• Step 1: Define DeviceAuth helper

Create backend/DeviceHub/DeviceAuth.cs:

using backend.Data;
using backend.Devices;
using Microsoft.EntityFrameworkCore;

namespace backend.DeviceHub;

public class DeviceAuth(AppDbContext db, DeviceTokenService tokens)
{
    public async Task<Device?> AuthenticateAsync(string? authHeader, CancellationToken ct)
    {
        if (string.IsNullOrEmpty(authHeader)) return null;
        const string prefix = "Bearer ";
        if (!authHeader.StartsWith(prefix, StringComparison.Ordinal)) return null;
        var token = authHeader[prefix.Length..].Trim();
        if (string.IsNullOrEmpty(token)) return null;
        var hash = tokens.Hash(token);
        var device = await db.Devices.Include(d => d.Config)
            .FirstOrDefaultAsync(d => d.TokenHash == hash && !d.IsRevoked, ct);
        return device;
    }
}

• Step 2: Implement the endpoint (handshake only)

Create backend/DeviceHub/DeviceHubEndpoint.cs:

using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
using backend.Data;
using backend.Devices;
using Microsoft.EntityFrameworkCore;

namespace backend.DeviceHub;

public static class DeviceHubEndpoint
{
    public static IEndpointRouteBuilder MapDeviceHub(this IEndpointRouteBuilder app)
    {
        app.Map("/device", async (HttpContext ctx, DeviceAuth auth, DeviceRegistry reg, AppDbContext db) =>
        {
            if (!ctx.WebSockets.IsWebSocketRequest)
            {
                ctx.Response.StatusCode = StatusCodes.Status400BadRequest;
                return;
            }

            var device = await auth.AuthenticateAsync(ctx.Request.Headers.Authorization.ToString(), ctx.RequestAborted);
            if (device is null)
            {
                ctx.Response.StatusCode = StatusCodes.Status401Unauthorized;
                return;
            }

            using var ws = await ctx.WebSockets.AcceptWebSocketAsync();
            await using var active = new ActiveDevice(device.Id, ws);
            if (!reg.TryRegister(active))
            {
                await ws.CloseAsync(WebSocketCloseStatus.PolicyViolation,
                    "device already connected", ctx.RequestAborted);
                return;
            }

            try
            {
                await HandleAsync(active, device, db, ctx.RequestAborted);
            }
            finally
            {
                reg.Unregister(device.Id);
            }
        });
        return app;
    }

    private static async Task HandleAsync(ActiveDevice active, Device device, AppDbContext db, CancellationToken ct)
    {
        var buffer = new byte[8192];
        var ws = ActiveWs(active);

        while (ws.State == WebSocketState.Open && !ct.IsCancellationRequested)
        {
            using var msg = new MemoryStream();
            WebSocketReceiveResult res;
            while (true)
            {
                res = await ws.ReceiveAsync(buffer, ct);
                if (res.MessageType == WebSocketMessageType.Close) return;
                msg.Write(buffer, 0, res.Count);
                if (res.EndOfMessage) break;
            }

            if (res.MessageType == WebSocketMessageType.Binary)
            {
                // Wake-path handling lives in Task 18. For now, drop.
                if (msg.Length != 3840) continue;
                continue;
            }

            var text = Encoding.UTF8.GetString(msg.ToArray());
            using var doc = JsonDocument.Parse(text);
            var type = doc.RootElement.GetProperty("type").GetString();
            switch (type)
            {
                case "hello":
                    await SendHelloAckAsync(active, device, ct);
                    break;

                case "ping":
                    device.LastSeenAt = DateTimeOffset.UtcNow;
                    await db.SaveChangesAsync(ct);
                    await active.SendEnvelopeAsync(new PongEnvelope("pong"), ct);
                    break;
            }
        }
    }

    private static WebSocket ActiveWs(ActiveDevice active)
    {
        // Reach into the private WS via reflection-free accessor: ActiveDevice exposes nothing else; we know the field.
        // To avoid touching internals, ActiveDevice gains a `WebSocket Ws { get; }` getter. (Update Task 16 file accordingly.)
        return active.Ws;
    }

    private static async Task SendHelloAckAsync(ActiveDevice active, Device device, CancellationToken ct)
    {
        var enabled = device.Config?.EnabledToolsJson ?? "[]";
        var config = JsonNode.Parse($$"""
        {
          "voice": "{{Escape(device.Config?.Voice)}}",
          "model": "{{Escape(device.Config?.Model)}}",
          "system_prompt": "{{Escape(device.Config?.SystemPrompt)}}",
          "idle_timeout_seconds": {{device.Config?.IdleTimeoutSeconds ?? 30}},
          "enabled_tools": {{enabled}}
        }
        """)!.AsObject();

        await active.WriteEnvelopeAsync(new JsonObject
        {
            ["type"] = "hello_ack",
            ["config"] = config,
        }, ct);
    }

    private static string Escape(string? s) =>
        (s ?? "").Replace("\\", "\\\\").Replace("\"", "\\\"");
}

Reflecting the note above: in backend/DeviceHub/ActiveDevice.cs, expose the WS:

public WebSocket Ws => _ws;

(Add this as a public property next to DeviceId.)

• Step 3: Wire it up

In backend/Program.cs:

builder.Services.AddScoped<backend.DeviceHub.DeviceAuth>();

After other endpoint registrations but BEFORE app.Run();, add:

app.UseWebSockets();
app.MapDeviceHub();

(UseWebSockets() must run before Map("/device") to enable WS upgrades.)

• Step 4: Write the integration test

Create backend.tests/DeviceHubHandshakeTests.cs:

using System.Net.Http.Json;
using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using backend.Devices;
using FluentAssertions;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

namespace backend.tests;

public class DeviceHubHandshakeTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    private async Task<string> PairAndGetTokenAsync()
    {
        var http = _factory.CreateClient();
        await http.PostAsJsonAsync("/api/auth/register",
            new { email = "ws@example.com", password = "Passw0rd!" });
        await http.PostAsJsonAsync("/api/auth/login",
            new { email = "ws@example.com", password = "Passw0rd!" });
        var codeRes = await http.PostAsJsonAsync("/api/pair-code", new { name = "ws-test" });
        var code = (await codeRes.Content.ReadFromJsonAsync<Dictionary<string, object>>())!["code"].ToString()!;

        var pairRes = await _factory.CreateClient().PostAsJsonAsync("/api/pair",
            new { code, hostname = "smoke", client_version = "0.1" });
        var pairBody = await pairRes.Content.ReadFromJsonAsync<Dictionary<string, object>>();
        return pairBody!["device_token"].ToString()!;
    }

    [Fact]
    public async Task Connect_without_token_returns_401()
    {
        var client = _factory.Server.CreateWebSocketClient();
        var uri = new Uri(_factory.Server.BaseAddress, "device");
        var act = async () => await client.ConnectAsync(uri, CancellationToken.None);
        await act.Should().ThrowAsync<Exception>(); // 401 surfaces as a WS upgrade error
    }

    [Fact]
    public async Task Hello_ack_includes_config()
    {
        var token = await PairAndGetTokenAsync();
        var client = _factory.Server.CreateWebSocketClient();
        client.ConfigureRequest = req => req.Headers.Authorization =
            new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);

        var uri = new Uri(_factory.Server.BaseAddress, "device");
        var ws = await client.ConnectAsync(uri, CancellationToken.None);

        await SendJson(ws, new { type = "hello", device_id = Guid.NewGuid(), client_version = "0.1" });
        var ack = await ReceiveJson(ws);
        ack.GetProperty("type").GetString().Should().Be("hello_ack");
        ack.GetProperty("config").GetProperty("voice").GetString().Should().NotBeNullOrEmpty();
        ack.GetProperty("config").GetProperty("model").GetString().Should().NotBeNullOrEmpty();
        ack.GetProperty("config").GetProperty("enabled_tools").GetArrayLength().Should().BeGreaterThan(0);
    }

    [Fact]
    public async Task Ping_replies_with_pong_and_updates_last_seen()
    {
        var token = await PairAndGetTokenAsync();
        var client = _factory.Server.CreateWebSocketClient();
        client.ConfigureRequest = req => req.Headers.Authorization =
            new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
        var uri = new Uri(_factory.Server.BaseAddress, "device");
        var ws = await client.ConnectAsync(uri, CancellationToken.None);

        await SendJson(ws, new { type = "hello", device_id = Guid.NewGuid(), client_version = "0.1" });
        await ReceiveJson(ws); // drain ack
        await SendJson(ws, new { type = "ping" });
        var pong = await ReceiveJson(ws);
        pong.GetProperty("type").GetString().Should().Be("pong");
    }

    private static async Task SendJson(WebSocket ws, object payload)
    {
        var bytes = Encoding.UTF8.GetBytes(JsonSerializer.Serialize(payload));
        await ws.SendAsync(bytes, WebSocketMessageType.Text, true, CancellationToken.None);
    }

    private static async Task<JsonElement> ReceiveJson(WebSocket ws)
    {
        var buf = new byte[8192];
        using var stream = new MemoryStream();
        while (true)
        {
            var r = await ws.ReceiveAsync(buf, CancellationToken.None);
            stream.Write(buf, 0, r.Count);
            if (r.EndOfMessage) break;
        }
        return JsonDocument.Parse(stream.ToArray()).RootElement.Clone();
    }
}

• Step 5: Run the tests

Run: dotnet test --filter DeviceHubHandshakeTests Expected: Passed: 3.

• Step 6: Commit

git add backend/DeviceHub/DeviceAuth.cs backend/DeviceHub/DeviceHubEndpoint.cs backend/DeviceHub/ActiveDevice.cs backend/Program.cs backend.tests/DeviceHubHandshakeTests.cs
git commit -m "DeviceHub: /device WS with bearer auth + hello/hello_ack + ping/pong"

---

Task 18: DeviceHubEndpoint — wake → session lifecycle, audio relay, tool_result fwd

Files:

• Modify: backend/DeviceHub/DeviceHubEndpoint.cs
• Create: backend.tests/DeviceHubWakeFlowTests.cs

Add to the dispatch:

• wake: if no active session, fetch DeviceConfig from the DB, build RealtimeSettings, ask RealtimeSessionFactory.CreateAsync(...), kick off session.RunAsync(...) in the background. Track the session on the ActiveDevice.
• Binary frames: route to the active session's WriteUplinkAsync.
• cancel: cancel the session CTS.
• playback_done: log only — used by the Pi state machine; no server logic needed in v1.
• tool_result: route to ActiveDevice.CompleteToolResult(...).

For the integration test, swap in a fake RealtimeSessionFactory that uses ScriptedRealtimeUpstream instead of OpenAI. Inject by overriding the DI registration in ApiFactory.

• Step 1: Extend ActiveDevice with session tracking

In backend/DeviceHub/ActiveDevice.cs, add:

public RealtimeSession? CurrentSession { get; set; }
public CancellationTokenSource? SessionCts { get; set; }

(Imports: using backend.Realtime;.)

• Step 2: Make RealtimeSessionFactory test-overridable

In backend/Realtime/RealtimeSessionFactory.cs, mark the CreateAsync method virtual so tests can subclass and inject ScriptedRealtimeUpstream.

Change the signature:

public virtual async Task<RealtimeSession> CreateAsync(...)

• Step 3: Extend the dispatch

In backend/DeviceHub/DeviceHubEndpoint.cs, after the existing using backend.Data;, add:

using backend.Realtime;
using backend.Tools;
using backend.Conversations;

Change the Map("/device", ...) lambda signature to also pull RealtimeSessionFactory and the DI scope provider:

app.Map("/device", async (
    HttpContext ctx,
    DeviceAuth auth,
    DeviceRegistry reg,
    AppDbContext db,
    RealtimeSessionFactory sessions,
    IServiceScopeFactory scopes) =>

Pass sessions, scopes down into HandleAsync (add to its parameters).

In the dispatch switch inside HandleAsync, add:

case "wake":
    if (active.CurrentSession is null)
        await StartSessionAsync(active, device, db, sessions, scopes, ct);
    break;

case "cancel":
    active.SessionCts?.Cancel();
    break;

case "playback_done":
    break; // v1 client coordination only

case "tool_result":
    var env = JsonSerializer.Deserialize<ToolResultEnvelope>(text)!;
    active.CompleteToolResult(env);
    break;

And in the binary-frame branch, replace the drop with:

if (msg.Length == 3840 && active.CurrentSession is not null)
{
    await active.CurrentSession.WriteUplinkAsync(msg.ToArray(), ct);
}

Add the helper:

private static async Task StartSessionAsync(
    ActiveDevice active, Device device, AppDbContext db,
    RealtimeSessionFactory sessions, IServiceScopeFactory scopes, CancellationToken parentCt)
{
    var enabledTools = JsonSerializer.Deserialize<string[]>(device.Config?.EnabledToolsJson ?? "[]")
        ?? Array.Empty<string>();
    var cfg = new RealtimeSettings(
        Model: device.Config?.Model ?? "gpt-4o-realtime-preview",
        Voice: device.Config?.Voice ?? "alloy",
        SystemPrompt: device.Config?.SystemPrompt ?? "",
        IdleTimeoutSeconds: device.Config?.IdleTimeoutSeconds ?? 30,
        EnabledTools: new HashSet<string>(enabledTools));

    active.SessionCts = CancellationTokenSource.CreateLinkedTokenSource(parentCt);
    var ct = active.SessionCts.Token;

    var sessionScope = scopes.CreateAsyncScope();
    var scopedSessions = sessionScope.ServiceProvider.GetRequiredService<RealtimeSessionFactory>();

    RealtimeSession session;
    try
    {
        session = await scopedSessions.CreateAsync(
            device.Id, cfg, active, active, ct);
    }
    catch (Exception ex)
    {
        await active.SendEnvelopeAsync(
            new ErrorEnvelope("error", "upstream", ex.Message, true), parentCt);
        await sessionScope.DisposeAsync();
        return;
    }

    active.CurrentSession = session;
    _ = Task.Run(async () =>
    {
        try { await session.RunAsync(ct); }
        finally
        {
            active.CurrentSession = null;
            active.SessionCts?.Dispose();
            active.SessionCts = null;
            await sessionScope.DisposeAsync();
        }
    }, ct);
}

(Pull using Microsoft.Extensions.DependencyInjection; to the top.)

• Step 4: Override the factory in tests

In backend.tests/ApiFactory.cs, add a field public ScriptedRealtimeUpstream? Upstream { get; set; }. Override services:

protected override void ConfigureWebHost(IWebHostBuilder builder)
{
    builder.UseEnvironment("Test");
    builder.ConfigureAppConfiguration((_, cfg) =>
    {
        cfg.AddInMemoryCollection(new Dictionary<string, string?>
        {
            ["ConnectionStrings:Default"] = $"Data Source={_dbPath}",
            ["OPENAI_API_KEY"] = "test-key",
        });
    });

    builder.ConfigureTestServices(services =>
    {
        services.AddScoped<RealtimeSessionFactory, FakeRealtimeSessionFactory>();
    });
}

Add using backend.Realtime; and using Microsoft.Extensions.DependencyInjection; to the file. Then create FakeRealtimeSessionFactory:

public class FakeRealtimeSessionFactory(
    OpenAIKeyProvider keys,
    ToolRegistry registry,
    ConversationLog log,
    ILoggerFactory loggerFactory)
    : RealtimeSessionFactory(keys, registry, log, loggerFactory)
{
    public static ScriptedRealtimeUpstream Upstream { get; } = new();

    public override Task<RealtimeSession> CreateAsync(
        Guid deviceId, RealtimeSettings cfg, ISessionOutput output,
        IDeviceChannel channel, CancellationToken ct)
    {
        Upstream.Push(new System.Text.Json.Nodes.JsonObject { ["type"] = "session.updated" });
        var s = new RealtimeSession(
            deviceId, Upstream, output, log, registry, cfg, channel,
            loggerFactory.CreateLogger<RealtimeSession>());
        return Task.FromResult(s);
    }
}

(Adjust the visibility of the protected ctor params of RealtimeSessionFactory if needed — or keep them on the base class as protected internal for the inheritance to compile. Easier: in RealtimeSessionFactory change the constructor params to a single protected ctor, and store them as protected fields the subclass uses.)

Update RealtimeSessionFactory:

public class RealtimeSessionFactory
{
    protected readonly OpenAIKeyProvider keys;
    protected readonly ToolRegistry registry;
    protected readonly ConversationLog log;
    protected readonly ILoggerFactory loggerFactory;

    public RealtimeSessionFactory(OpenAIKeyProvider keys, ToolRegistry registry,
        ConversationLog log, ILoggerFactory loggerFactory)
    {
        this.keys = keys; this.registry = registry; this.log = log; this.loggerFactory = loggerFactory;
    }

    public virtual async Task<RealtimeSession> CreateAsync(...) // unchanged body
}

• Step 5: Write the integration test

Create backend.tests/DeviceHubWakeFlowTests.cs:

using System.Net.Http.Json;
using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
using FluentAssertions;
using Xunit;

namespace backend.tests;

public class DeviceHubWakeFlowTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
    private readonly ApiFactory _factory = factory;

    private async Task<string> PairAndGetTokenAsync(string email)
    {
        var http = _factory.CreateClient();
        await http.PostAsJsonAsync("/api/auth/register",
            new { email, password = "Passw0rd!" });
        await http.PostAsJsonAsync("/api/auth/login",
            new { email, password = "Passw0rd!" });
        var codeRes = await http.PostAsJsonAsync("/api/pair-code", new { name = "wake-test" });
        var code = (await codeRes.Content.ReadFromJsonAsync<Dictionary<string, object>>())!["code"].ToString()!;
        var pairRes = await _factory.CreateClient().PostAsJsonAsync("/api/pair",
            new { code, hostname = "smoke", client_version = "0.1" });
        var pairBody = await pairRes.Content.ReadFromJsonAsync<Dictionary<string, object>>();
        return pairBody!["device_token"].ToString()!;
    }

    [Fact]
    public async Task Wake_opens_session_started_then_assistant_done_then_session_ended_on_idle()
    {
        var token = await PairAndGetTokenAsync("wake1@example.com");
        var client = _factory.Server.CreateWebSocketClient();
        client.ConfigureRequest = req => req.Headers.Authorization =
            new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
        var uri = new Uri(_factory.Server.BaseAddress, "device");
        var ws = await client.ConnectAsync(uri, CancellationToken.None);

        await SendJson(ws, new { type = "hello", device_id = Guid.NewGuid(), client_version = "0.1" });
        await ReceiveJson(ws); // ack

        await SendJson(ws, new { type = "wake", at = 0L });
        var started = await ReceiveJson(ws);
        started.GetProperty("type").GetString().Should().Be("session_started");

        FakeRealtimeSessionFactory.Upstream.Push(new JsonObject
        {
            ["type"] = "response.audio_transcript.done",
            ["transcript"] = "hello!",
        });
        FakeRealtimeSessionFactory.Upstream.Push(new JsonObject { ["type"] = "response.done" });

        var done = await ReceiveJson(ws);
        done.GetProperty("type").GetString().Should().Be("assistant_done");

        // Send cancel to end the session quickly (don't wait for idle-timeout).
        await SendJson(ws, new { type = "cancel" });
        var ended = await ReceiveJson(ws);
        ended.GetProperty("type").GetString().Should().Be("session_ended");
    }

    [Fact]
    public async Task Binary_uplink_after_wake_reaches_upstream_as_input_audio_buffer_append()
    {
        var token = await PairAndGetTokenAsync("wake2@example.com");
        var client = _factory.Server.CreateWebSocketClient();
        client.ConfigureRequest = req => req.Headers.Authorization =
            new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
        var uri = new Uri(_factory.Server.BaseAddress, "device");
        var ws = await client.ConnectAsync(uri, CancellationToken.None);

        await SendJson(ws, new { type = "hello", device_id = Guid.NewGuid(), client_version = "0.1" });
        await ReceiveJson(ws);
        await SendJson(ws, new { type = "wake", at = 0L });
        await ReceiveJson(ws); // session_started

        var frame = new byte[3840];
        for (int i = 0; i < frame.Length; i++) frame[i] = 0x55;
        await ws.SendAsync(frame, WebSocketMessageType.Binary, true, CancellationToken.None);

        await Task.Delay(200);
        FakeRealtimeSessionFactory.Upstream.Sent
            .Any(e => (string?)e["type"] == "input_audio_buffer.append")
            .Should().BeTrue();

        await SendJson(ws, new { type = "cancel" });
        await ReceiveJson(ws);
    }

    private static async Task SendJson(WebSocket ws, object payload)
    {
        var bytes = Encoding.UTF8.GetBytes(JsonSerializer.Serialize(payload));
        await ws.SendAsync(bytes, WebSocketMessageType.Text, true, CancellationToken.None);
    }

    private static async Task<JsonElement> ReceiveJson(WebSocket ws)
    {
        var buf = new byte[16384];
        using var stream = new MemoryStream();
        while (true)
        {
            var r = await ws.ReceiveAsync(buf, CancellationToken.None);
            if (r.MessageType == WebSocketMessageType.Binary) { stream.Write(buf, 0, r.Count); continue; }
            stream.Write(buf, 0, r.Count);
            if (r.EndOfMessage) break;
        }
        return JsonDocument.Parse(stream.ToArray()).RootElement.Clone();
    }
}

• Step 6: Run the tests

Run: dotnet test --filter DeviceHubWakeFlowTests Expected: Passed: 2.

• Step 7: Run the full suite

Run: dotnet test Expected: every test (Plan 1 + everything above) passes.

• Step 8: Commit

git add backend/DeviceHub backend/Realtime/RealtimeSessionFactory.cs backend.tests/ApiFactory.cs backend.tests/DeviceHubWakeFlowTests.cs
git commit -m "DeviceHub: wake → session lifecycle + uplink relay + tool_result routing"

---

Task 19: Coolify deploy + live smoke

Files:

• Modify: README.md

Coolify already runs the backend image (from Plan 1). Plan 2 only added code; no Dockerfile change needed. Trigger a deploy via the Coolify v4 API and confirm /device rejects unauthenticated WS upgrades on the live host.

• Step 1: Ensure OPENAI_API_KEY is set on the Coolify app

Use the deploying-to-coolify-via-api skill. If the env var is not already present on the app, PATCH it via the API. Do NOT log the value.

If unsure, ask the user to confirm the key is set and proceed once they do.

• Step 2: Trigger a deploy (fire-and-forget per global CLAUDE.md)

curl -sf -X POST \
  -H "Authorization: Bearer $COOLIFY_KEY" \
  "$COOLIFY_URL/api/v1/deploy?uuid=$APP_UUID"

Read APP_UUID from deploy.json if present, otherwise from the Coolify app list. Expected: a 2xx response with a deployment UUID. Print it and stop — do NOT poll the build.

• Step 3: Live smoke — /device rejects unauthenticated WS upgrade

After a few minutes (or whenever the user signals the deploy is live), confirm the endpoint is reachable. From the sandbox:

DOMAIN="https://assistant.volcanic.tes.gd"
# Just check the HTTP-level handshake fails closed.
curl -sI -H "Connection: Upgrade" -H "Upgrade: websocket" \
     -H "Sec-WebSocket-Version: 13" \
     -H "Sec-WebSocket-Key: $(head -c16 /dev/urandom | base64)" \
     "$DOMAIN/device" | head -1

Expected: a 401 Unauthorized status line (no Authorization header was sent).

• Step 4: Live smoke — authenticated handshake works

Re-use the Plan 1 smoke recipe to pair a "smoke" device, then connect with the returned token using a tiny Python script. From the sandbox:

DOMAIN="https://assistant.volcanic.tes.gd"
COOKIE=$(mktemp)
curl -sf -c "$COOKIE" -X POST "$DOMAIN/api/auth/login" \
  -H 'content-type: application/json' \
  -d '{"email":"<your existing email>","password":"<your password>"}'
CODE=$(curl -sf -c "$COOKIE" -b "$COOKIE" -X POST "$DOMAIN/api/pair-code" \
  -H 'content-type: application/json' \
  -d '{"name":"plan2-smoke"}' | grep -oE '"code":"[^"]+"' | cut -d'"' -f4)
TOKEN=$(curl -sf -X POST "$DOMAIN/api/pair" \
  -H 'content-type: application/json' \
  -d "{\"code\":\"$CODE\",\"hostname\":\"smoke\",\"client_version\":\"0.1\"}" \
  | grep -oE '"device_token":"[^"]+"' | cut -d'"' -f4)
echo "TOKEN length: ${#TOKEN}"

python3 - <<PY
import asyncio, json, sys
import websockets
URL = "wss://assistant.volcanic.tes.gd/device"
TOKEN = "$TOKEN"

async def main():
    async with websockets.connect(URL, additional_headers={"Authorization": f"Bearer {TOKEN}"}) as ws:
        await ws.send(json.dumps({"type":"hello","device_id":"00000000-0000-0000-0000-000000000000","client_version":"0.1"}))
        ack = json.loads(await ws.recv())
        print("ack:", ack["type"])
        assert ack["type"] == "hello_ack"
        await ws.send(json.dumps({"type":"ping"}))
        pong = json.loads(await ws.recv())
        print("pong:", pong["type"])
        assert pong["type"] == "pong"
        print("OK")
asyncio.run(main())
PY

If websockets isn't available in the sandbox, install with pip install --user websockets. Expected output: ack: hello_ack then pong: pong then OK.

(Do NOT send wake from the smoke — it would open a real OpenAI session and burn budget for no return. The Plan 3 client exercises that end-to-end.)

• Step 5: Update README with Plan 2 surface area

Append the following section to README.md:

## Plan 2: device hub + Realtime relay

- `wss://<domain>/device` — bearer-token WebSocket. Pair a device first to get a token.
- Handshake: client sends `{type:"hello", device_id, client_version}`, backend replies `hello_ack` with per-device config.
- Heartbeat: client sends `{type:"ping"}` every 15 s, backend replies `pong` and updates `Devices.LastSeenAt`.
- Wake: `{type:"wake"}` opens a Realtime session against OpenAI; backend emits `session_started`, then forwards `response.audio.delta` as binary, `response.audio_transcript.done` → assistant turn, etc. On `end_session` tool, idle timeout, or upstream error, emits `session_ended`.
- Tools shipped: `get_current_time` (server-side), `end_session` (closes session), `set_volume` (round-trips to Pi).

The Python Pi client (Plan 3) is the production consumer of this surface. The handshake-only smoke recipe in this README is enough to confirm the deploy is alive without paying OpenAI.

• Step 6: Commit

git add README.md
git commit -m "README: document Plan 2 device-hub surface area"

---

Self-review notes

• Spec coverage:

  • DeviceHub WS + auth → Tasks 16, 17, 18
  • Realtime relay (open / session.update / audio relay / transcripts / tools / end_session / idle) → Tasks 9–14
  • Tools registry + 3 built-ins → Tasks 3, 4, 5
  • Conversations + Turns data model + log → Tasks 1, 2
  • OpenAI key from env → Task 6 + Coolify env var (Task 19)
  • Backend → device wire envelopes (hello_ack, session_started, assistant_done, session_ended, tool_call, error) → Task 16
  • Pi-side tool round-trip (tool_call/tool_result) → Tasks 5, 16, 18
  • Idle timeout reset on speech_started → Task 14
  • end_session sequencing (close only after response.done) → Tasks 12, 13
  • Conversation EndReason persisted → Tasks 1, 2, 9
  • Deploy fire-and-forget via Coolify v4 API → Task 19

• Deferred (explicit, per the design spec section "Open items deferred to implementation plans"):

  • Final voice / model defaults remain as currently seeded in SystemSettings; pick a different default by updating that singleton row in Plan 4's admin UI work.
  • play_audio push-from-server flow has envelope types (HubEnvelopes) but no production path; activate in v1.1 when per-device audio is added.
  • The Python client itself (client/) is still the Plan 1 placeholder; full implementation in Plan 3.

• Type / name consistency:

  • RealtimeSettings fields: Model, Voice, SystemPrompt, IdleTimeoutSeconds, EnabledTools — used the same in Tasks 7, 9, 10, 11, 12, 14, 15, 18.
  • ToolResult.Output is a JsonElement — preserved across Tasks 3, 4, 5, 12.
  • IDeviceChannel.CallPiToolAsync(name, args, ct) signature is identical in Tasks 3, 5, 16.
  • RealtimeSession ctor argument order matches between Task 9, Task 15 factory, and Task 18 fake factory.
  • ConversationLog methods: StartAsync, AppendUserAsync, AppendAssistantAsync, AppendToolAsync, EndAsync — same set used in Tasks 2, 11, 12, 13.

• Plan 3 and 4 hand-off:

  • Plan 3 (Python client) implements the wire protocol defined in spec § "Wire protocol" and Task 16's envelope set. It is the integration test for everything here.
  • Plan 4 (UI) adds the DeviceConfig edit endpoint + an admin push of config_updated. The relay already takes effect on the NEXT wake (it reads DeviceConfig fresh in Task 18 StartSessionAsync), so the design constraint "changes take effect on the next session" is already honoured without any extra wiring needed in Plan 2.