If you want an AI agent to listen to a phone call and talk back, you need the raw audio. Not a recording after the fact, not a transcript, the actual PCM samples while the call is happening, and a way to push samples back so the agent can speak. That is what a media gateway does: it sits between the carrier and your application and moves audio frames in real time.
We built one for Handset. This is how it works and the two or three decisions that turned out to matter.
Two faces, one protocol
The gateway has two WebSocket faces and they have almost nothing in common.
The carrier face is dialed by the provider. When a stream starts on a call, Telnyx (in live mode) or our simulated carrier (in test mode) opens a socket to us and pushes audio in its own format: its own JSON shape, its own field names, its own base64 encoding of the samples. Providers differ here, so the frames arriving on this socket are provider-specific.
The customer face is connected by your application, and it must be identical no matter which carrier is underneath. Your speech loop should not know or care whether the audio originated at Telnyx or a simulator. So the gateway translates: a per-provider dialect decodes the carrier frames into one neutral internal shape, and a single Handset frame protocol goes out to you.
The customer protocol is deliberately boring. Text frames, JSON, base64-encoded 8 kHz PCMU, with a sequence number and a millisecond timestamp:
{"event":"stream.started","stream_id":"stm_…","call_id":"call_…",
"direction":"fork","track":"both","codec":"pcmu","sample_rate":8000}
{"event":"media","track":"inbound","seq":42,"timestamp_ms":840,
"payload":"<base64 8kHz PCMU>"}
{"event":"stream.stopped","reason":"call_ended"}For bidirectional streams you send the same media frame back and it is played into the call, plus a clear to flush anything queued:
{"event":"media","payload":"<base64 8kHz PCMU>"}
{"event":"clear"}We chose JSON with base64 over a binary framing for v1 on purpose. At 8 kHz PCMU the payload is small, every language can parse it without a codec library, and you can read a stream in your terminal while debugging. When someone needs 16 kHz stereo the calculus changes, but shipping the debuggable version first was the right call.
Two secrets for two very different sockets
The faces authenticate differently because the threats are different.
The carrier dials us from its own infrastructure and has no way to hold a bearer token, so the carrier URL is HMAC-signed with an expiry. The provider gets a URL with exp and sig query params and a fmt telling the gateway which dialect to decode; the gateway recomputes the MAC and checks the clock. No signature or an expired one, no socket.
wss://media.handset.dev/carrier/stm_…?exp=1725400000&sig=…&fmt=telnyx
The customer connects from code you control, so it uses a one-time bearer token (hsms_…). We store only its SHA-256 hash on the stream row and show the token exactly once; presenting it connects you and burns it. A leaked token from a log is worthless after first use, and there is no long-lived credential sitting in the stream record.
Both keys derive from the same encryption key the rest of the platform already uses, so standing up the gateway added no new secret to provision or rotate.
Buffering: drop the oldest, never block
Audio is only useful in real time. A frame that arrives 400 ms late is worse than a frame that never arrives, because late audio pushes everything behind it later still. So each direction has a bounded buffer, about five seconds of frames, and the enqueue never blocks:
func pushDropOldest(ch chan []byte, b []byte) {
for {
select {
case ch <- b:
return
default:
select {
case <-ch: // buffer full: discard the oldest, try again
default:
}
}
}
}If your consumer stalls, you lose the oldest audio and stay current, rather than backing pressure all the way up to the carrier socket and stalling the call for everyone. Preserving real-time behavior beats preserving every sample. This is the opposite of what you want for a recording, and exactly what you want for a live loop.
The hard part: a teardown across two processes
Here is the bug that shaped the design. The stream socket lives in the API process. But a call ends in a different process: the hangup arrives as a carrier webhook that a background worker handles. When the worker marks the call ended, it has no handle on the WebSocket, it is not even in the same process. Carrier commands do not cross process boundaries. So how does the socket in process A learn that process B ended the call?
The answer is that the database row is the bus. The streams row has a status that goes starting → active → stopped. Anything that ends a stream, the hangup worker, an explicit DELETE, a shutdown, settles that row. And the gateway session runs a small watchdog that polls its own row every three seconds:
// Watchdog: the row leaving 'active' is the cross-process stop signal.
var status, reason string
g.Store.Pool.QueryRow(ctx,
`SELECT status, COALESCE(stop_reason,'') FROM streams WHERE id = $1`,
row.ID).Scan(&status, &reason)
if status != "starting" && status != "active" {
s.finalize(reason) // row left active: tear the sockets down
}No message bus, no pub/sub, no extra infrastructure. The row we already have to keep is the coordination point. Whoever settles it wins, and the socket-holding process notices within three seconds and tears down cleanly, sending the stream.stopped frame as its protocol-level goodbye.
Exactly-once teardown
Because more than one thing can trigger the end, the teardown has to be idempotent. Finalize runs at most once per session (a sync.Once), and the billing write is guarded by the row's own status:
UPDATE streams SET status = 'stopped', stop_reason = $2, stopped_at = now()
WHERE id = $1 AND status IN ('starting','active')
RETURNING started_at, stopped_atIf the API's DELETE already flipped the row to stopped, the watchdog's finalize updates zero rows and bills nothing. The WHERE status IN (...) is the whole concurrency story: the first writer to move the row out of an active state is the one that bills, and every later attempt is a no-op. We use this exact shape all over the platform, and it is the single most useful pattern in the codebase.
One clock, or you will bill wrong
Streaming is billed per minute, so duration has to be right. The obvious implementation, note the time when the socket opens and subtract when it closes, gave us durations that were wrong by nearly a second. The gateway process ran on a VM whose clock was 0.87 seconds behind the database. started_at came from the Go process, stopped_at from Postgres now(), and subtracting across two clocks produced nonsense.
The fix is a rule, not a patch: billing durations are computed from one clock only, and it is the database's. Both activate and finalize set their timestamps with Postgres now() and return them, so the subtraction is always between two readings of the same clock:
UPDATE streams SET status = 'active', started_at = now()
WHERE id = $1 AND status = 'starting'
RETURNING started_atThe Go process still keeps a monotonic clock for frame timestamps, but those are relative offsets within a stream and never touch money. Anything you charge for gets its endpoints from the same source.
Testing it without a phone
You cannot iterate on any of this if every test costs a phone call. The way out is the fake carrier we wrote about earlier: in test mode, the simulator dials the gateway's carrier face exactly like Telnyx would, signed URL and dialect and all, and pushes synthesized audio. The gateway cannot tell the difference, which means the entire path above, the HMAC check, the dialect decode, the buffering, the watchdog, the exactly-once finalize, runs in a unit test with no network and no carrier. Every decision in this post has a test that exercises it against a stream that never left the machine.
What it is for
The gateway is plumbing, but it is the plumbing everything voice-and-AI sits on. Live transcription reads the inbound track off it. Agent-assist watches that transcript mid-call. The browser softphone is a customer-face client. And a voice agent is a bidirectional stream: inbound audio to your speech-to-text, your text-to-speech back into the call as media frames.
If you are building anything that needs to hear or speak on a live call, the streaming reference is at docs.handset.dev, and there is a working app on the real API, no signup, at demo.handset.dev.
More from Handset