Streaming model responses without a janky UI
A client in Noida has a compliance dashboard where analysts review generated summaries of regulatory filings. The previous vendor piped the model's output through a single fetch request and waited for the full response. Summaries run 800-1200 tokens. On a good day the request took 9 seconds, during which the UI showed a spinner and nothing else. Analysts started opening a second tab and pasting the filing into ChatGPT directly.
We switched to streaming, and the first version of the streaming UI was worse than the spinner. Here is what went wrong and what we fixed.
The model provider sends Server-Sent Events. The browser's EventSource API handles the transport, but it gives you chunks that arrive mid-token, mid-word, mid-sentence. If you append each chunk directly to the DOM, you get content that flickers and reflows as markdown partially parses. A chunk arriving with `## Impo` followed by `rtant` renders a header, then un-renders it, then renders it again. We buffer chunks on the client and flush to the DOM every 80ms using a timer, not on every chunk arrival. The user sees smooth text instead of stuttering.
Markdown parsing is the second problem. You cannot run a full markdown parser on every flush because partial markdown is ambiguous — an unclosed code block swallows everything after it. We keep two representations: a raw buffer for the source text and a rendered node for display. On each flush we parse the buffer with a parser that tolerates incomplete input. We use marked with a patch that closes unclosed fences before parsing and strips the artificial closing fence from the output. The alternative is rendering raw text until the stream completes, but analysts copy from the rendered output, so raw text was a non-starter.
Cancellation is the part most teams skip. An analyst reads the first paragraph, realises the model is summarising the wrong section, and wants to stop. We add a Cancel button that calls AbortController on the fetch and resets the UI. The edge case is the chunk in flight: the abort fires, but the last SSE event is already parsed and queued in the event loop. We tag every render with the request ID from the response header and discard renders whose ID does not match the active request. Without this, the stale chunk appends after the cancellation and the user sees a phantom sentence appear on a cleared screen.
The reconnect logic in EventSource is another trap. EventSource reconnects automatically after a drop, but it starts a new request from the beginning. For a streaming completion that means the user sees the summary restart from the top. We disable EventSource's auto-reconnect and use fetch with a ReadableStream instead. On disconnect we show a Retry button that sends the last received token count as a continuation offset. Not all providers support this, so for the ones that do not, we restart and accept the duplicate. We surface the restart in the UI with a small "reconnected" label rather than hiding it.
The cost angle is real. Streaming does not change the total token count, but it changes the failure mode. A request that times out at 7 seconds with a spinner costs nothing. A streaming request that delivers 400 tokens and then drops has already been billed for those tokens. For a product doing 12,000 summaries a month at roughly 600 tokens each, partial-failure waste adds about 3% to the model bill. We track it with a simple counter in the response handler.
What we did not do: build a custom SSE parser. The browser's streaming primitives are good enough. The jank is in the render loop and the state machine around cancellation, not in the transport.