Ollama

Ollama is a free, open-source tool that lets you run large language models (LLMs) locally on your own machine — no cloud, no API keys, no data leaving your computer.

Core Idea

Instead of calling OpenAI/Anthropic APIs, you download and run models directly:

ollama run llama3.2
# → pulls the model, starts a chat in your terminal

That’s it. A full LLM running locally.

What It Does

• Downloads and manages models from a model registry
• Serves a local REST API (compatible with OpenAI’s API format)
• Handles all the complexity of quantization, GPU layers, memory management
• Runs on Mac, Linux, and Windows

Hardware Support

Apple Silicon Macs are particularly well-suited because of unified memory — a MacBook Pro with 32GB RAM can run surprisingly capable models.

Model Library

Ollama hosts a registry at ollama.com/library. Popular models include:

Models are quantized (compressed) to fit consumer hardware — e.g., a 7B model typically needs ~4–8GB of RAM/VRAM.

CLI Commands

# Run a model (downloads if not present)
ollama run llama3.2
# Pull a model without running it
ollama pull mistral
# List installed models
ollama list
# Remove a model
ollama rm llama3.2
# Show model info
ollama show llama3.2
# Run a specific quantization
ollama run llama3.2:8b-instruct-q5_K_M
# Serve the API (runs automatically, but can be explicit)
ollama serve

REST API

Ollama exposes a local API on port 11434:

# Generate (streaming)
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain Loki in one sentence"
}'
# Chat (OpenAI-compatible)
curl http://localhost:11434/v1/chat/completions -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

The OpenAI-compatible endpoint (/v1/...) means you can drop Ollama into any app that uses the OpenAI SDK by just changing the base URL.

Using with OpenAI SDK

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # required by SDK, value doesn't matter
)
response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Modelfile — Custom Models

You can create custom models with a Modelfile, similar to a Dockerfile:

FROM llama3.2
# Set system prompt
SYSTEM """
You are a helpful DevOps assistant who specializes in 
Kubernetes, Prometheus, and Grafana Loki.
"""
# Set parameters
PARAMETER temperature 0.7
PARAMETER num_ctx 4096

ollama create devops-assistant -f Modelfile
ollama run devops-assistant

Integrations

Ollama works with a huge ecosystem:

Ollama vs Alternatives

Common Use Cases

• Privacy-first AI — sensitive data never leaves your machine
• Offline use — works without internet after model download
• Local RAG — pair with a vector DB for document Q&A
• Development/testing — prototype without API costs
• Self-hosted AI tools — run your own Copilot, chatbot, etc.

Quick Setup

# macOS
brew install ollama
# Linux
curl -fsSL https://ollama.com/install.sh | sh
# Then run your first model
ollama run llama3.2

Key Takeaways

1. Easiest way to run LLMs locally
2. OpenAI-compatible API — drop-in for many existing tools
3. Great on Apple Silicon — unified memory is a big advantage
4. Model quality has exploded — modern 7B models are genuinely useful
5. Privacy by default — nothing leaves your machine