Generative AI Part 6: How to Run LLMs Locally with Ollama

What you should know first: read Part 5: Temperature, Top-p and Controlling Model Output and Part 3: Your First LLM API Calls. You only need a terminal and a laptop with at least 8 GB of RAM.

So far in this series every model we used lived in someone else’s data centre: you sent a prompt over the internet, paid per token and trusted the provider with your data. In this part you will learn how to run LLMs locally with Ollama, a free open-source tool that downloads open models such as Llama, Gemma and Qwen and runs them on your own computer. You will install Ollama, pull your first model, chat with it in the terminal, call it from Python and Node.js, stream answers, reuse your OpenAI code from Part 3 and even build your own custom model with a Modelfile. No API key, no bill, and it works offline.

Why run LLMs locally?

In Part 3 you called OpenAI, Claude and Gemini over the internet. Those models are the smartest available, but every request costs money, needs an API key and sends your text to another company. A local LLM flips all of that. The model file sits on your disk and runs on your own CPU or GPU.

The trade-off is quality and speed. A 3-billion-parameter model on a laptop will not match the biggest cloud models at hard reasoning, but it is surprisingly good at summaries, simple Q&A, rewriting text and code help. Many real apps use both: a local model for cheap, private tasks and a cloud model for the hardest ones.

What is Ollama and how it works

Ollama is a free, open-source app for macOS, Windows and Linux that makes local models as easy as Docker made containers. It does three jobs for you:

1. Downloads models from its library (ollama.com/library) in a compressed (quantized) format called GGUF, so an 8B model needs about 5 GB instead of 16 GB.
2. Runs a local server on http://localhost:11434 that loads the model into RAM or GPU memory and answers requests.
3. Gives you simple tools: a command-line chat, a REST API, official Python and JavaScript packages and an OpenAI-compatible endpoint.

Install Ollama and run your first model

On macOS and Windows, download the installer from the official site and open it; Ollama then runs in the background (you will see its icon in the menu bar or system tray). On Linux, use the install script. Then pull a model and start chatting:

# macOS / Windows: download the installer from https://ollama.com/download
# Linux: one-line install script
$ curl -fsSL https://ollama.com/install.sh | sh
$ ollama --version
# download a small model (about 2 GB) and chat with it
$ ollama pull llama3.2
$ ollama run llama3.2
# type a message, press Enter; type /bye to quit

The first ollama run takes a few seconds while the model loads into memory; after that, answers start almost instantly. If you skip ollama pull, ollama run downloads the model for you automatically. Here are the everyday commands you will use:

$ ollama list            # models on your disk
$ ollama ps              # models loaded in memory right now
$ ollama run llama3.2 "Explain an API in one sentence"   # one-shot prompt
$ ollama show llama3.2   # size, context length, parameters
$ ollama stop llama3.2   # unload it from memory
$ ollama rm llama3.2     # delete it from disk

Choosing a model for your computer

The most common beginner mistake is downloading a model that is too big. A simple rule: a model compressed to 4 bits needs roughly 0.6–0.7 GB of memory per billion parameters, plus extra for the context window. So a 3B model needs about 2–3 GB, an 8B model about 5–6 GB and a 32B model about 20 GB. Click your setup below for safe picks:

Model names change quickly as new versions arrive, so treat these as starting points and browse the Ollama model library for the newest options. Look at the size tag and the “tools”, “vision” or “thinking” labels on each model page.

The local REST API

Everything Ollama does goes through its HTTP server, so any language that can send a request can use a local model. The two main endpoints are /api/generate (single prompt) and /api/chat (a list of messages with roles, exactly like Part 4). Set "stream": false to get one JSON reply instead of many small chunks:

# Ollama runs a local REST API on port 11434
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{ "role": "user", "content": "Say hello to DevDojo readers" }],
  "stream": false,
  "options": { "temperature": 0.2 }
}'

$ bash api_chat.sh
{"model":"llama3.2","created_at":"…","message":{"role":"assistant","content":"Hello, DevDojo readers! Happy coding and keep learning!"},"done":true,"done_reason":"stop","total_duration":…,"prompt_eval_count":…,"eval_count":…}

Click ▶ Run to see the shape of the JSON reply. The … parts are timing and token numbers that differ on every machine.

Notice that the reply has the same role and content shape you saw with cloud APIs, plus useful stats: prompt_eval_count (input tokens), eval_count (output tokens) and total_duration in nanoseconds.

Run LLMs locally with Ollama from Python and Node.js

For real apps, use the official ollama packages. They wrap the REST API with nice types, and the code looks almost the same in both languages. Install them first (and keep Ollama running in the background):

# Python
$ pip install -U ollama openai
# Node.js (ES modules, Node 18+)
$ npm install ollama openai

Now send a chat request with a system message, a user message and the sampling options you learned in Part 5. num_ctx sets the context window size in tokens and num_predict is the local version of “max tokens”:

# chat_local.py - talk to a local model with the official ollama package
from ollama import chat

response = chat(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "You are a friendly coding tutor. Answer in 2 short sentences."},
        {"role": "user", "content": "What is a REST API?"},
    ],
    options={"temperature": 0.2, "top_p": 0.9, "seed": 42, "num_ctx": 4096, "num_predict": 120},
)

print(response.message.content)
print("tokens in:", response.prompt_eval_count, "| tokens out:", response.eval_count)
print(f"time: {response.total_duration / 1e9:.1f}s")

$ python chat_local.py
A REST API is a way for programs to talk to each other over HTTP using URLs and methods like GET and POST. For example, an app can send GET /users to a server and receive the list of users as JSON.
tokens in: <n> | tokens out: <n>
time: <seconds>s

// chat-local.mjs - talk to a local model with the official ollama package
import ollama from "ollama";

const response = await ollama.chat({
  model: "llama3.2",
  messages: [
    { role: "system", content: "You are a friendly coding tutor. Answer in 2 short sentences." },
    { role: "user", content: "What is a REST API?" },
  ],
  options: { temperature: 0.2, top_p: 0.9, seed: 42, num_ctx: 4096, num_predict: 120 },
});

console.log(response.message.content);
console.log("tokens in:", response.prompt_eval_count, "| tokens out:", response.eval_count);
console.log(`time: ${(response.total_duration / 1e9).toFixed(1)}s`);

$ node chat-local.mjs
A REST API is a way for programs to talk to each other over HTTP using URLs and methods like GET and POST. For example, an app can send GET /users to a server and receive the list of users as JSON.
tokens in: <n> | tokens out: <n>
time: <seconds>s

Switch between the Python and Node.js tabs, then click ▶ Run to see the output.

Streaming answers

Local models write a few words at a time. Instead of waiting for the full answer, stream it so users see text appear instantly (we will build a full chat UI with streaming in Part 7). Pass stream=True and loop over the chunks:

# stream_local.py - print the answer word by word as the model writes it
from ollama import chat

stream = chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Give me 3 short tips to learn Python."}],
    stream=True,
)

for chunk in stream:
    print(chunk.message.content, end="", flush=True)
print()

$ python stream_local.py
1. Practise a little every day, even 20 minutes.
2. Build small projects like a to-do app or a quiz.
3. Read other people's code and run it yourself.

// stream-local.mjs - print the answer word by word as the model writes it
import ollama from "ollama";

const stream = await ollama.chat({
  model: "llama3.2",
  messages: [{ role: "user", content: "Give me 3 short tips to learn Python." }],
  stream: true,
});

for await (const part of stream) {
  process.stdout.write(part.message.content);
}
process.stdout.write("\n");

$ node stream-local.mjs
1. Practise a little every day, even 20 minutes.
2. Build small projects like a to-do app or a quiz.
3. Read other people's code and run it yourself.

Click ▶ Run; in a real terminal the text appears word by word.

Reuse OpenAI code with a local model

Already have code written with the OpenAI SDK from Part 3? Ollama speaks the same language at /v1. Change the base_url, pass any API key and use a local model name. This is a great way to develop and test for free, then switch to a cloud model in production by changing two lines:

# openai_local.py - reuse your OpenAI code from Part 3 with a local model
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",  # Ollama's OpenAI-compatible API
    api_key="ollama",                      # required by the SDK, ignored by Ollama
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain JSON to a beginner in one sentence."}],
    temperature=0.2,
)

print(response.choices[0].message.content)

$ python openai_local.py
JSON is a simple text format for storing data as key-value pairs, like {"name": "Asha", "age": 21}, that almost every programming language can read.

// openai-local.mjs - reuse your OpenAI code from Part 3 with a local model
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11434/v1", // Ollama's OpenAI-compatible API
  apiKey: "ollama",                     // required by the SDK, ignored by Ollama
});

const response = await client.chat.completions.create({
  model: "llama3.2",
  messages: [{ role: "user", content: "Explain JSON to a beginner in one sentence." }],
  temperature: 0.2,
});

console.log(response.choices[0].message.content);

$ node openai-local.mjs
JSON is a simple text format for storing data as key-value pairs, like {"name": "Asha", "age": 21}, that almost every programming language can read.

Same OpenAI SDK, local model. Compare the two tabs with your Part 3 code.

Build a custom model with a Modelfile

Typing the same system prompt and settings every time gets boring. A Modelfile is like a Dockerfile for models: it starts FROM a base model and saves your own defaults and personality under a new name. This does not train the model; it packages a prompt and settings so every app gets the same behaviour.

# Modelfile - build your own "DevDojo Tutor" on top of llama3.2
FROM llama3.2

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

SYSTEM """
You are DevDojo Tutor. Explain coding ideas to beginners
in simple English, with one short code example.
"""

$ ollama create devdojo-tutor -f Modelfile
gathering model components
using existing layer sha256:…
creating new layer sha256:…
writing manifest
success

Click ▶ Run to see what ollama create prints.

$ ollama run devdojo-tutor "What is a variable?"
# it also works from code: model="devdojo-tutor"
$ ollama list

Common errors and fixes

Best practices

• Start small, scale up. Begin with a 1–4B model, check that answers are good enough, and only then try a bigger one.
• Pin model tags in your code (for example llama3.2:3b), so an update to the default tag does not silently change your app.
• Set a seed and low temperature for tests, so the same prompt gives the same answer on your machine.
• Keep the server private. Port 11434 has no login, so never expose it to the internet without a proxy and authentication.
• Use the OpenAI-compatible API plus environment variables, so you can switch between local and cloud models easily.
• Check the licence of each open model before using it in a paid product.
• Watch disk space. Models are big; remove old ones with ollama rm.

Practice task

Build a tiny “local model showdown”:

1. Pull two models that fit your RAM (for example llama3.2:1b and llama3.2).
2. Extend chat_local.py to loop over both models and ask the same three questions with seed=42 and temperature=0.2.
3. Print each answer with eval_count and speed in tokens per second (eval_count / (eval_duration / 1e9)).
4. Create a Modelfile for a “Git helper” that only answers Git questions, then test it with ollama run.
5. Point your OpenAI code from Part 3 at Ollama using environment variables and run it without an API key.

FAQ: running LLMs locally with Ollama

Conclusion

You can now run LLMs locally with Ollama: install it, pull an open model that fits your computer, chat in the terminal, call it from Python and Node.js, stream answers, reuse your OpenAI code by changing the base URL and package your own assistant with a Modelfile. Local models will not replace the biggest cloud models for every task, but they are perfect for learning, privacy-sensitive work and free experiments. For every command and option, keep the official Ollama documentation handy.

Missed the previous part? Read Part 5: Temperature, Top-p and Controlling Model Output. Want to build something bigger with LLMs? Try our RAG tutorial with Python and pgvector.