Generative AI Part 7: How to Stream LLM Responses and Build a Simple Chat UI

What you should know first: read Part 6: Run LLMs Locally with Ollama and Part 3: Your First LLM API Calls. You need Node.js 20.6+ or Python 3.10+, and either a free local model in Ollama or an OpenAI API key.

When you ask ChatGPT a question, the answer does not appear all at once. It types itself out, word by word. That trick is called streaming, and it is the single biggest thing that makes an AI app feel fast. In this part you will learn how to stream LLM responses in Python and Node.js, look at the raw Server-Sent Events that travel over the wire, build a terminal chatbot that remembers the conversation and then build a simple chat UI in the browser with a Stop button. Every example works with a free local model through Ollama or with a cloud model such as OpenAI, by changing just one setting.

What is streaming?

An LLM writes its answer one token at a time (you learned about tokens in Part 2). In a normal API call, the provider waits until the very last token is ready, then sends the complete answer in one response. With streaming, the provider sends each new token, or a small group of tokens, the moment it exists. Your code receives a stream of tiny pieces and can show them right away.

Think of ordering food. A normal call is like waiting at the counter until the whole meal is packed. Streaming is like a restaurant that serves each dish as soon as it leaves the kitchen: the total time is similar, but you start eating much sooner.

Why stream LLM responses?

The one thing to remember: streaming improves time to first token (how quickly something appears), not the total time. If you need the full answer before doing anything, for example to parse JSON, a normal call is often simpler. We will cover reliable JSON in the next part.

Your first streamed answer in Python and Node.js

Let us start with the cloud example from Part 3 and make it stream. Set up a folder and install the packages first. If you followed Part 3 you already have most of this.

# 1. project folder (Node.js 20.6+ and Python 3.10+)
$ mkdir devdojo-chat && cd devdojo-chat && mkdir public
$ npm init -y && npm pkg set type=module
$ npm install openai
$ pip install openai fastapi uvicorn python-dotenv
# 2a. free and local: make sure Ollama is running (Part 6)
$ ollama pull llama3.2
# 2b. or cloud: put these lines in a .env file instead
# LLM_BASE_URL=https://api.openai.com/v1
# LLM_API_KEY=sk-your-openai-key
# LLM_MODEL=gpt-6-luna

Now the code. Compared with Part 3 there is only one new line, stream: true. Instead of one response object you get an iterator of events. Each event has a type. The two you care about most are response.output_text.delta (a new piece of text in event.delta) and response.completed (the end, with token usage).

# pip install openai python-dotenv
import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI()  # reads OPENAI_API_KEY from .env
model = os.getenv("OPENAI_MODEL", "gpt-6-luna")

stream = client.responses.create(
    model=model,
    instructions="You are a friendly coding teacher.",
    input="Explain streaming in one short sentence.",
    stream=True,  # <- the only change from Part 3
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)  # print each piece now
    elif event.type == "response.completed":
        usage = event.response.usage
        print(f"\n[done] tokens out: {usage.output_tokens}")

$ python stream_basic.py
Streaming means the model sends its answer in small pieces as soon as they are ready.
[done] tokens out: 18

// npm install openai
// run with: node --env-file=.env stream-basic.mjs
import OpenAI from "openai";

const client = new OpenAI(); // reads OPENAI_API_KEY
const model = process.env.OPENAI_MODEL ?? "gpt-6-luna";

const stream = await client.responses.create({
  model,
  instructions: "You are a friendly coding teacher.",
  input: "Explain streaming in one short sentence.",
  stream: true, // <- the only change from Part 3
});

for await (const event of stream) {
  if (event.type === "response.output_text.delta") {
    process.stdout.write(event.delta); // print each piece now
  } else if (event.type === "response.completed") {
    console.log(`\n[done] tokens out: ${event.response.usage.output_tokens}`);
  }
}

$ node --env-file=.env stream-basic.mjs
Streaming means the model sends its answer in small pieces as soon as they are ready.
[done] tokens out: 18

Switch between the Python and Node.js tabs and click ▶ Run to see what prints. In your terminal the sentence appears a few words at a time.

Two small details make it look right in a terminal. In Python, print(..., end="", flush=True) prints the piece without a new line and pushes it to the screen immediately. In Node.js, process.stdout.write() does the same. Without them the text would appear in big jumps or all at the end.

What travels over the wire: Server-Sent Events

SDKs hide the details, but it helps to see the raw stream once. LLM APIs use a simple web standard called Server-Sent Events (SSE). The server keeps one HTTP response open and writes lines that start with data:, each followed by a blank line. Ollama speaks the same OpenAI-compatible format, so you can try it locally with curl:

# -N turns off curl's buffering so you see chunks as they arrive
curl -N http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2","stream":true,
       "messages":[{"role":"user","content":"Say hi"}]}'

$ bash raw_stream.sh
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hi"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":" there"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}]}

data: [DONE]

Click ▶ Run to see the raw stream. Every data: line is one small JSON chunk, and [DONE] marks the end.

Each chunk carries a delta: only the new text, not the whole answer so far. Your job is to join the deltas together. The last chunk has a finish_reason such as "stop" (finished normally) or "length" (hit the token limit), and [DONE] closes the stream.

The openai package gives you two styles of streaming. The newer Responses API (client.responses.create) sends named events, as you saw above. The older Chat Completions API (client.chat.completions.create) sends the chat.completion.chunk objects shown here. For the rest of this post we use Chat Completions, because Ollama, LM Studio, vLLM, Groq and many other providers support that exact format. One codebase then works with local and cloud models.

A terminal chatbot with memory

A chatbot is a loop: read the question, stream the answer, repeat. The only tricky part is memory. The model forgets everything between calls, so we keep a messages list and send the whole list every time. After the stream ends, we add the assistant’s full reply to the list.

# A terminal chatbot that streams answers and remembers the conversation.
# Works with Ollama (default) or any OpenAI-compatible API.
import os
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL", "http://localhost:11434/v1"),
    api_key=os.getenv("LLM_API_KEY", "ollama"),  # Ollama ignores the key
)
MODEL = os.getenv("LLM_MODEL", "llama3.2")

messages = [{"role": "system", "content": "You are a helpful, concise assistant."}]

while True:
    user_text = input("\nYou: ").strip()
    if user_text.lower() in {"exit", "quit"}:
        break
    messages.append({"role": "user", "content": user_text})

    print("AI: ", end="", flush=True)
    reply = ""
    stream = client.chat.completions.create(
        model=MODEL, messages=messages, stream=True
    )
    for chunk in stream:
        if not chunk.choices:
            continue
        piece = chunk.choices[0].delta.content or ""
        reply += piece
        print(piece, end="", flush=True)

    # save the full answer so the next question has context
    messages.append({"role": "assistant", "content": reply})
    print()

$ python chat_cli.py

You: What is Node.js in one line?
AI: Node.js is a runtime that lets you run JavaScript outside the browser.

You: And who made it?
AI: Node.js was created by Ryan Dahl in 2009.

You: exit

Click ▶ Run. Notice that the second question (“And who made it?”) works only because the history is sent again.

By default this talks to Ollama on your own computer (http://localhost:11434/v1). To use OpenAI instead, set LLM_BASE_URL, LLM_API_KEY and LLM_MODEL as shown in the setup block. Nothing else changes. Also notice if not chunk.choices: continue: some providers send a final chunk with usage numbers and no choices, and this line stops it from crashing your loop.

Build a simple chat UI in the browser

Now the fun part: a real chat page with bubbles that fill up word by word. We need two pieces. A tiny server that holds the API key, calls the LLM with streaming and writes each piece into the HTTP response. And a single HTML page that sends the conversation and reads the response as a stream.

The server below uses only Node.js built-ins plus the openai package, so there is no framework to learn. Read the comments, then switch to the public/index.html tab.

// Tiny chat server: node --env-file=.env server.mjs  ->  http://localhost:3000
import http from "node:http";
import { readFile } from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: process.env.LLM_BASE_URL ?? "http://localhost:11434/v1",
  apiKey: process.env.LLM_API_KEY ?? "ollama",
});
const MODEL = process.env.LLM_MODEL ?? "llama3.2";
const MAX_MESSAGES = 20; // keep the history (and the bill) small

const server = http.createServer(async (req, res) => {
  if (req.method === "GET" && req.url === "/") {
    res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
    return res.end(await readFile("public/index.html"));
  }
  if (req.method === "POST" && req.url === "/api/chat") {
    let body = "";
    for await (const part of req) body += part;
    const { messages = [] } = JSON.parse(body || "{}");

    res.writeHead(200, {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "no-cache",
      "X-Accel-Buffering": "no", // stop Nginx from buffering the stream
    });

    const controller = new AbortController();
    res.on("close", () => controller.abort()); // user left or pressed Stop

    try {
      const stream = await client.chat.completions.create(
        {
          model: MODEL,
          stream: true,
          messages: [
            { role: "system", content: "You are a friendly coding teacher." },
            ...messages.slice(-MAX_MESSAGES),
          ],
        },
        { signal: controller.signal }
      );
      for await (const chunk of stream) {
        res.write(chunk.choices[0]?.delta?.content ?? "");
      }
    } catch (err) {
      if (!controller.signal.aborted) res.write("\n[error] " + err.message);
    }
    return res.end();
  }
  res.writeHead(404).end("Not found");
});

server.listen(3000, () => console.log("Chat UI on http://localhost:3000"));

$ node --env-file=.env server.mjs
Chat UI on http://localhost:3000

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>DevDojo Chat</title>
  <style>
    body { font-family: system-ui; max-width: 640px; margin: 40px auto; }
    #log p { padding: 10px 14px; border-radius: 12px; white-space: pre-wrap; }
    .user { background: #e8f0ff; margin-left: 20%; }
    .ai { background: #f2f2f2; margin-right: 20%; }
  </style>
</head>
<body>
  <h1>DevDojo Chat</h1>
  <div id="log"></div>
  <form id="form">
    <input id="text" placeholder="Ask anything..." autocomplete="off" required>
    <button id="send">Send</button>
    <button id="stop" type="button" disabled>Stop</button>
  </form>
  <script>
    const messages = []; // the whole conversation lives here
    const log = document.getElementById("log");
    let controller = null;

    function addBubble(role, text) {
      const p = document.createElement("p");
      p.className = role === "user" ? "user" : "ai";
      p.textContent = text;
      log.append(p);
      return p;
    }

    document.getElementById("form").onsubmit = async (e) => {
      e.preventDefault();
      const input = document.getElementById("text");
      messages.push({ role: "user", content: input.value });
      addBubble("user", input.value);
      input.value = "";

      const bubble = addBubble("assistant", "");
      controller = new AbortController();
      stop.disabled = false;

      try {
        const res = await fetch("/api/chat", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ messages }),
          signal: controller.signal,
        });
        const reader = res.body.getReader();
        const decoder = new TextDecoder();
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          bubble.textContent += decoder.decode(value, { stream: true });
        }
      } catch (err) {
        if (err.name !== "AbortError") bubble.textContent += " [error]";
      }
      messages.push({ role: "assistant", content: bubble.textContent });
      stop.disabled = true;
    };

    const stop = document.getElementById("stop");
    stop.onclick = () => controller?.abort(); // cancel the answer
  </script>
</body>
</html>

Two files: click the server.mjs and public/index.html tabs. Click ▶ Run on the server tab, then open http://localhost:3000 in your browser.

How the browser reads the stream

The key lines are in the page’s <script>. A normal fetch call would use await res.json(), which waits for the whole body. Instead we call res.body.getReader(). Each reader.read() returns the next bytes as soon as they arrive, and TextDecoder turns bytes into text. The { stream: true } option on decode matters: a Hindi or emoji character can be split across two chunks, and this option keeps the half-character until the rest arrives.

The server sends plain text rather than SSE. That keeps the browser code tiny, because we do not need to parse data: lines. SSE is great when you need events with names (for example “tool started”), but for a basic chat, plain text streaming is enough. If you want to understand readable streams deeper, MDN has an excellent guide on using readable streams.

Open the step explorer below to follow one answer from the Send button to the screen.

The same chat server in Python with FastAPI

Prefer Python? FastAPI has StreamingResponse, which takes a generator. Every yield sends one piece to the browser. We use AsyncOpenAI so the server can handle many chats at the same time without blocking. The HTML page stays exactly the same.

# pip install fastapi uvicorn openai
# run with: uvicorn main:app --reload
import os
from fastapi import FastAPI
from fastapi.responses import FileResponse, StreamingResponse
from openai import AsyncOpenAI
from pydantic import BaseModel

client = AsyncOpenAI(
    base_url=os.getenv("LLM_BASE_URL", "http://localhost:11434/v1"),
    api_key=os.getenv("LLM_API_KEY", "ollama"),
)
MODEL = os.getenv("LLM_MODEL", "llama3.2")
app = FastAPI()


class ChatRequest(BaseModel):
    messages: list[dict]


@app.get("/")
async def home():
    return FileResponse("public/index.html")  # same page as the Node.js version


@app.post("/api/chat")
async def chat(req: ChatRequest):
    async def generate():
        stream = await client.chat.completions.create(
            model=MODEL,
            stream=True,
            messages=[{"role": "system", "content": "You are a friendly coding teacher."}]
            + req.messages[-20:],
        )
        async for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content

    return StreamingResponse(
        generate(),
        media_type="text/plain; charset=utf-8",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

$ uvicorn main:app --port 3000
INFO:     Started server process […]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:3000 (Press CTRL+C to quit)

Click ▶ Run. The FastAPI server serves the exact same public/index.html, so the browser code does not change.

If you are new to FastAPI, our FastAPI tutorial explains routes, Pydantic models and Uvicorn step by step.

Stopping an answer with a Stop button

Long answers cost money and time, so a Stop button is a must. Our page already has one. Here is the chain of events when you click it:

The text that already arrived stays in the bubble and is still saved to messages, so the conversation keeps going naturally. In FastAPI, Starlette stops the generator when the client disconnects, which also closes the upstream stream.

Common errors and fixes

Best practices for streaming chat apps

• Keep secrets on the server. The browser talks to your API route; your route talks to the LLM.
• Use textContent, not innerHTML, when you show model output. Model text can contain HTML, and inserting it raw opens the door to XSS attacks.
• Limit the history (for example the last 20 messages) and set max_tokens or max_output_tokens so one chat cannot burn your budget.
• Always offer Stop and abort the upstream request when the client disconnects.
• Show a typing hint (a blinking cursor or “…”) until the first piece arrives, so users know something is happening.
• Handle errors inside the stream. Once you have sent a 200 status you cannot change it, so write a short error message into the stream, as our server does.
• Turn off buffering in proxies and hosting platforms, and test the deployed app, not just localhost.
• Render Markdown later. Models often reply in Markdown. Once the basics work, render it with a safe Markdown library that sanitises HTML.

Practice task

Frequently asked questions

Conclusion

You now know how to stream LLM responses end to end: turn on stream: true, loop over events or chunks, join the deltas, and forward them from a small server to a browser that reads the response with getReader(). You built a terminal chatbot and a simple chat UI with memory and a Stop button, and the same code works with a free local model in Ollama or with a cloud model. This is the foundation of almost every AI chat product you use.

Missed the previous part? Read Part 6: Run LLMs Locally with Ollama. Want to build something bigger? Our RAG tutorial with Python and pgvector shows how to chat with your own documents.