Streaming vs Batched Edges

A Beach channel is either streaming or batched. The interior of your application doesn't know which — that's the whole point of the channel-agnostic interior — but the outbound edge does, because the choice changes what the edge subscribes to and how it ships the reply.

This guide is the decision and the implications.

The distinction

Property Streaming edge Batched edge
Latency to first visible output Sub-second Seconds to minutes
Granularity Each part as it lands One settled artefact
Underlying transport Persistent connection (SSE, WebSocket, voice) One-shot send (SMTP, SMS gateway, push)
What it subscribes to The actor's part stream A Delivery Manifest's onComplete
Interim respond() parts Streamed live Discarded
Examples Chat panel, voice, live ops dashboard Email, SMS, push notification

The orchestrator doesn't know

The orchestrator emits respond() calls with parts. It may emit several across iterations of the tool loop — { partType: 'thinking', text: 'checking availability...' }, then { partType: 'response', text: 'I found three options' }. It declares completion via turnState.

A streaming edge subscribes to the part stream and flushes each part to the client as it arrives. The user sees thinking → response progressively. A batched edge subscribes to nothing per-part — it holds open a Delivery Manifest, waits for the orchestrator's final respond() to settle, and ships once.

The orchestrator emits the same parts in both cases. The edge decides what to do with them.

Streaming edges

When you have one

  • A web chat panel, mobile app, voice assistant, IDE plugin.
  • Any UI where the user is actively waiting and visible "I'm working on it" feedback is part of the UX.
  • Live operator surfaces (a dashboard a human is watching).

What the wiring looks like

The Beach starter's chatCollector is the canonical streaming edge. It publishes each parts array to a transport (Redis pub/sub by default) keyed on sessionId. The consumer's HTTP endpoint subscribes and forwards.

import { registerCanonicalHandlers } from '@cool-ai/beach-starter';

registerCanonicalHandlers(router, {
  // … other config …
  streamingChannels: ['sse'],
  chatPublish: async (sessionId, parts) => {
    await redis.publish(`reply:${sessionId}`, JSON.stringify(parts));
  },
});

// HTTP side
app.get('/events', async (req, res) => {
  const { sessionId } = req.query;
  res.setHeader('Content-Type', 'text/event-stream');
  const sub = new Redis();
  await sub.subscribe(`reply:${sessionId}`);
  sub.on('message', (_ch, payload) => res.write(`data: ${payload}\n\n`));
  req.on('close', () => sub.disconnect());
});

Each respond() call from the orchestrator becomes one SSE chunk. Interim "thinking" parts flush instantly. The browser renders them and replaces them as final content arrives.

Properties

  • Sub-second to first byte. As soon as the orchestrator says anything, the user sees it.
  • No buffering. Mistakes are visible. If the orchestrator emits a part it later wants to retract, it can't — the user already saw it.
  • Connection-bound. Streaming requires a persistent connection. If the user's network drops, the next chunk goes nowhere. Reconnection is the consumer's problem (resume from a lastEventId, or just reconnect and re-fetch the session state).
  • One actor instance to one connection. The starter's SSE adapter stores one ServerResponse per sessionId; calling connect() twice replaces the first. If your application has multiple browser tabs per user, that's an application-level decision (one shared session vs separate sessions).

Batched edges

When you have one

  • Email — even fast email is "deliver in seconds, not milliseconds."
  • SMS, WhatsApp messages, push notifications.
  • Any "fire one message and stop" outbound.
  • Any channel where the cost of a partial send is high (SMS character limits, email subject lines that can't be amended).

What the wiring looks like

A batched edge opens a Delivery Manifest when the inbound message arrives, runs the orchestrator's turn to settlement, fills the manifest's main_reply slot from the settled parts, and ships in the manifest's onComplete.

import { Manifest, ManifestRegistry } from '@cool-ai/beach-core';

const manifestRegistry = new ManifestRegistry();

// Inside your inbound handler — pseudocode, see Setting Up Email for a full example
async function onInboundEmail(missive) {
  await missiveStore.write(missive);

  const manifestId = `email-delivery:${missive.id}`;
  const manifest = new Manifest({
    id: manifestId,
    expected: ['main_reply'],
    timeoutMs: 5 * 60_000,
    onComplete: async (filled) => {
      const artifact = await formatter.format({ inbound: missive, filledSlots: filled });
      await emailChannel.send({
        to: artifact.to,
        subject: artifact.subject,
        text: artifact.text,
        html: artifact.html,
        inReplyTo: artifact.inReplyTo,
        references: artifact.references,
      });
    },
  });
  manifestRegistry.register(manifest);

  const settled = await sessionManager.runTurn({ /* … */ });
  manifestRegistry.deliver(manifestId, 'main_reply', { parts: settled.parts });
}

Interim respond() parts the orchestrator emits during its tool loop are produced and discarded. The Delivery Manifest only fills when the orchestrator declares turnState: 'complete'.

Properties

  • Latency is bounded by the slowest specialist. Email, SMS, etc. don't ship until the orchestrator's full reasoning is done. If a research specialist takes 30 seconds, the email arrives 30 seconds after the inbound.
  • One coherent message. The user sees a complete reply, not a stream of partial thoughts. This is usually what you want for email — partial thoughts in email are confusing.
  • Composite outbound supported. Batched outbound can wait for multiple slots (main_reply, subject_override, attachments) before sending. See Manifests.
  • Timeouts matter. A turn that hangs forever blocks the email outbound forever. Always set timeoutMs on the Delivery Manifest with a fallback path in onTimeout (send a "we're looking into this" reply, escalate to human, etc.).

Mixing — one application, both kinds

A common shape: the same orchestrator handles chat (streaming) and email (batched). Same tool loop, same prompts, same audit trail.

registerCanonicalHandlers(router, {
  orchestratorHandler: 'concierge',
  resolveSession: (data) => /* per-channel logic */,
  streamingChannels: ['sse'],
  batchedChannels: ['email'],
  chatPublish: redisPublish,
  store: missiveStore,
});

The orchestrator emits the same parts in both cases. The starter's replyDispatcher looks at event.data.channelId, decides which edge owns this reply, and routes:

  • channelId === 'sse' → streaming:deliver → chatCollector → SSE → browser.
  • channelId === 'email' → batched:deliver → responseCollector → missive store → outbound.

Two channels, one orchestrator. The orchestrator never reads channelId.

What if my channel is "kind of both"?

Some channels feel like they could be either. Three real cases:

Voice

Streaming. The user is on a call; latency to first audio matters. Use a streaming edge that synthesises speech as parts arrive.

Push notifications with a "tap to see more" surface

The notification itself is batched (one message). The surface that opens when tapped is streaming (a chat panel). Two edges, both backed by the same orchestrator.

Long-running batch reports ("daily digest")

Batched. There's no user actively watching; even if you streamed, no one would see it. Cron-trigger inbound, run the orchestrator, fill a Delivery Manifest with main_reply (the digest), ship via email or whatever.

If you find yourself wanting "streaming for the first 5 seconds, then stop and ship the rest as one message" — you have two channels. Wire both.

Pitfalls

Streaming an email body. SMTP isn't a streaming protocol. There's no "send progressively" — the message ships once with a complete body. Trying to bolt "streaming" onto email means buffering the parts, which is just batched with extra ceremony.

Batching a chat reply. The user sees nothing for 8 seconds while the orchestrator works, then a wall of text appears. They thought the bot was broken. Use a streaming edge — at least let the user see "thinking" parts.

Forgetting the Delivery Manifest's onTimeout. A specialist that hangs blocks the batched outbound forever. The user gets nothing. Always set timeoutMs and decide what to send (or who to alert) on timeout.

Mixing edge logic into the orchestrator. The orchestrator that does if (channelId === 'email') return shortReply; else return longReply; is fundamentally broken — it's pretending to be channel-agnostic while reading channel state. Either the work needs an edge-positioned actor (Composer specialist) or the channels need different orchestrators. See Reference: design principles, principle 2.7.

Trying to retract a streamed part. Once a chunk is on the wire, the user has it. The orchestrator can emit a follow-up part that supersedes it (and a smart UI can replace the prior render), but the user did briefly see the old content. Plan for it.

Related