Run LLMs Locally with Ollama: Complete Setup and API Guide

Cloud LLMs are convenient but expensive, rate-limited, and send your data to third parties. Ollama makes local LLMs as easy as running any other dev tool.

Installation

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Or via Homebrew
brew install ollama

# Windows: download from ollama.com

Running Models

# Pull and run a model (interactive chat)
ollama run llama3.2

# Available models
ollama run mistral
ollama run codellama
ollama run phi4
ollama run gemma3

# List downloaded models
ollama list

# Remove a model
ollama rm llama3.2

REST API

Ollama runs a local server at http://localhost:11434:

# Chat completion
curl http://localhost:11434/api/chat -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Explain async/await in JavaScript"}],
    "stream": false
}'

# Generate (completion mode)
curl http://localhost:11434/api/generate -d '{
    "model": "llama3.2",
    "prompt": "Write a Python function to reverse a string",
    "stream": false
}'

Python Integration

import requests

def chat(model: str, message: str, system: str = None) -> str:
    messages = []
    if system:
        messages.append({"role": "system", "content": system})
    messages.append({"role": "user", "content": message})

    response = requests.post(
        "http://localhost:11434/api/chat",
        json={"model": model, "messages": messages, "stream": False}
    )
    return response.json()["message"]["content"]

result = chat("llama3.2", "What is pgvector?")
print(result)

Or use the OpenAI-compatible API:

from openai import OpenAI

# Ollama is OpenAI-compatible
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Any non-empty string
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain reentrancy attacks"}]
)
print(response.choices[0].message.content)

This means you can swap Ollama for OpenAI or vice versa with a single line change.

Streaming Responses

import json

def stream_chat(model: str, message: str):
    response = requests.post(
        "http://localhost:11434/api/chat",
        json={"model": model, "messages": [{"role": "user", "content": message}], "stream": True},
        stream=True
    )

    for line in response.iter_lines():
        if line:
            chunk = json.loads(line)
            if not chunk.get("done"):
                print(chunk["message"]["content"], end="", flush=True)
    print()

stream_chat("llama3.2", "Write a Python web scraper")

Model Selection Guide

Models below 8B need at least 8GB RAM. 70B models need 64GB+ RAM or a powerful GPU.

Custom Modelfile

Customize a model's system prompt and parameters:

# Modelfile
FROM llama3.2

SYSTEM """
You are an expert Solidity developer. You write gas-efficient, secure smart contracts.
When reviewing code, always check for reentrancy, integer overflow, and access control issues.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

ollama create solidity-expert -f Modelfile
ollama run solidity-expert

When to Use Local vs Cloud

Use Ollama locally when:

• Processing sensitive/private data (medical records, legal documents, PII)
• High volume with predictable traffic (avoid rate limits)
• Offline or air-gapped environments
• Prototyping (no API costs)

Use cloud LLMs (Claude, GPT-4) when:

• Need best-in-class quality (Llama 8B is good, not GPT-4 level)
• Unpredictable traffic spikes
• Need multimodal capabilities (vision, audio)
• Team doesn't have the hardware

Ollama is excellent for internal tools, privacy-sensitive apps, and cost optimization at scale.