Securing Docker containers running AI tools (such as LiteLLM, Ollama, LocalAI, or custom inference runtimes) requires a defense-in-depth approach spanning container hardening, network isolation, credential management, and resource governance.

1. Enforce Container Hardening & Least Privilege

• Run as a Non-Root User: Never allow AI containers to execute as root inside the container. Specify a non-root UID/GID (e.g., user: "1000:1000") in your Docker Compose or Dockerfile.
• Make the Root Filesystem Read-Only: Use read_only: true to prevent attackers or compromised AI applications from modifying system binaries or installing malware. Mount a dedicated, isolated /tmp volume for transient model caching or scratch space.
• Drop Linux Kernel Capabilities: Strip unnecessary privileges by adding cap_drop: [ALL]. AI containers typically only need user-space execution.
• Prevent Privilege Escalation: Set security_opt: ["no-new-privileges:true"] to stop processes inside the container from gaining additional privileges via setuid binaries.

2. Isolate Network Access & API Interfaces

• Bind to Localhost (127.0.0.1): Unless the AI tool must be directly reachable across the local network, explicitly bind container ports to 127.0.0.1:PORT instead of exposing 0.0.0.0:PORT.
• Put Services Behind an Authenticated Reverse Proxy: Route public or team access through Nginx, Traefik, or Caddy configured with TLS/SSL, robust authentication (OAuth2/OIDC), and rate limiting.
• Restrict Outbound Egress: Use Docker custom networks or host firewall rules (UFW/iptables) to limit outbound internet access. If your AI tool processes sensitive data locally using local models (e.g., Ollama), block all outbound internet traffic for that container.

3. Implement Strict Secret Management & Access Controls

• Never Hardcode API Keys: Store external model keys (OpenAI, Anthropic, HuggingFace tokens) in environment variables loaded via .env files (secured with chmod 600) or dedicated secret managers (Vault, Docker Secrets).
• Enforce Virtual Keys and Spend Caps: If using AI proxies like LiteLLM, generate individual virtual API keys for each developer or team rather than sharing the master key. Assign daily/monthly financial budgets and rate limits per key.
• Redact Sensitive Data (PII/Credentials): Deploy guardrail middleware (such as Microsoft Presidio or custom regex filters) to automatically scrub PII, API tokens, and internal database connection strings before requests reach LLM providers.

4. Apply Resource Limits & Host Protection

• Set CPU and RAM Constraints: Heavy LLM workloads can quickly starve host system resources. Set explicit limits in Docker Compose to prevent Denial of Service (DoS):
  YAMLdeploy: resources: limits: cpus: '2.0' memory: 4096M
• Restrict GPU Access (If Applicable): When exposing NVIDIA GPUs to containers (nvidia-docker), pass only specific GPU IDs to dedicated containers rather than granting full access across all hardware.
• Monitor Metrics: Scrape container resource utilization (via cAdvisor + Prometheus) to detect abnormal compute spikes or unauthorized long-running batch processes.

5. Secure Image Supply Chain & Updates

• Use Official & Minimal Base Images: Use minimal base images (e.g., Alpine, Distroless, or official vendor images like ghcr.io/berriai/litellm). Avoid unverified community images from public registries.
• Pin Specific Version Tags: Never use the :latest image tag in production or staging environments. Pin explicit version tags (e.g., v1.59.0) or image digest hashes (@sha256:...) to prevent untested upstream changes.
• Vulnerability Scanning: Scan container images for CVEs using tools like Trivy, Grype, or Docker Scout before deploying them.

Here is a secure, production-ready docker-compose.yml for LiteLLM that implements local network binding, explicit CPU/memory constraints, non-root user execution, read-only container filesystems, and strict environment variable handling.

1. Project Directory Structure

Set up your project directory layout as follows before starting the container:

Plaintext

litellm-project/
├── .env                  # Private secrets (DO NOT COMMIT)
├── config.yaml           # LiteLLM routing & proxy settings
└── docker-compose.yml    # Hardened container stack

2. File Configurations

.env (Secrets File)

Create a .env file in the same directory. This keeps sensitive API keys isolated from your repository code.

Ini, TOML

# Upstream LLM API Keys
OPENAI_API_KEY=sk-proj-your-actual-openai-api-key-here
ANTHROPIC_API_KEY=sk-ant-your-actual-anthropic-api-key-here
# LiteLLM Administration Key (Used for generating virtual keys and proxy admin)
LITELLM_MASTER_KEY=sk-master-litellm-secret-key-change-this

config.yaml (LiteLLM Model & Security Config)

Define your backend models and security settings:

YAML

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  # Enable database / budget features if using LiteLLM database mode

docker-compose.yml (Hardened Deployment)

YAML

version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-v1.59.0
    container_name: litellm-proxy
    restart: unless-stopped
    
    # 1. Local Network Binding (Prevents exposure on public/LAN interfaces)
    ports:
      - "127.0.0.1:4000:4000"
    # 2. Command Execution with Config Mapping
    command: [ "--config", "/app/config.yaml", "--port", "4000", "--num_workers", "2" ]
    # 3. Mount Config & Read-Only Temp Volume
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - tmp_data:/tmp
    # 4. Environment Variables loaded from .env
    env_file:
      - .env
    # 5. Security & Isolation Enhancements
    read_only: true               # Prevents malicious writes to the container filesystem
    user: "1000:1000"             # Runs as a non-root user
    security_opt:
      - no-new-privileges:true    # Blocks privilege escalation attacks
    cap_drop:
      - ALL                       # Drops all Linux kernel capabilities
    # 6. Resource Limits (Prevents container from starving host CPU/RAM)
    deploy:
      resources:
        limits:
          cpus: '1.5'
          memory: 1536M
        reservations:
          cpus: '0.25'
          memory: 256M
    # 7. Health Check
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 15s
volumes:
  tmp_data:
    driver: local

Step-by-Step Deployment Instructions

1.Set Secure File Permissions for Secrets:

Ensure the .env file is accessible only by your host user:

chmod 600 .env

Verification: Run ls -l .env and confirm the permission bit shows -rw-------.

2.Start the LiteLLM Proxy Stack:

Launch the container in detached mode:

docker compose up -d

Verification: Run docker compose ps to verify the container state is Up (healthy).

3.Test Local Binding Security:

Verify that the proxy endpoints respond on 127.0.0.1 using your configured master key:

Bash

curl -i http://127.0.0.1:4000/health/readiness \
  -H "Authorization: Bearer sk-master-litellm-secret-key-change-this"

Verification: Expect an HTTP/1.1 200 OK response with {"status": "connected"}.

Securing open-source AI tools like LiteLLM, Ollama, LocalAI, or LangChain requires protecting three main attack vectors: API key exposure, network access, and resource depletion/model abuse.

Key Security Risks in Local AI Tools

1. API Key Theft & Model Misuse: Tools like LiteLLM proxy requests to upstream providers (OpenAI, Anthropic, local models). If the proxy is exposed or unauthenticated, bad actors can drain your API credits.
2. Prompt Injection & Data Leaks: Unfiltered AI inputs can leak internal database contents, source code, or system prompts.
3. Resource Exhaustion (DoS): Heavy LLM inferences can easily pin host CPUs/GPUs and crash local development servers.

Best Practices for Securing LiteLLM and Open-Source AI Stacks

1. Secure API Keys & Access Controls

• Never hardcode secrets in code or Docker files: Pass API keys into containers using environment files (.env) listed in .gitignore or secret managers.
• Enable Master Keys & Virtual Keys in LiteLLM: Require a master key for administration and issue scoped virtual keys to developers with spend/rate limits:
  YAML# config.yaml for LiteLLM Proxy model_list: - model_name: gpt-4o litellm_params: model: gpt-4o api_key: os.environ/OPENAI_API_KEY general_settings: master_key: sk-1234-your-secure-master-key
• Enforce Developer Spend Limits: Use LiteLLM’s built-in budget controls to set daily/monthly maximum spend per developer key.

2. Network Isolation & Access Boundaries

• Bind Services to Localhost Only: If tools like LiteLLM or Ollama are only for local host development, bind their ports to 127.0.0.1 instead of 0.0.0.0 to avoid exposing them on the LAN:
  Bash# Bind LiteLLM proxy port to localhost only docker run -p 127.0.0.1:4000:4000 ghcr.io/berriai/litellm:main-latest --config /app/config.yaml
• Restrict Inbound Network Rules: If hosted on a shared development server (like cvm-srv30), isolate the AI proxy port behind a firewall (UFW/iptables) so only authorized developer IPs can access port 4000.

3. Implement Input/Output Guardrails

• Use PII Anonymization: Configure LiteLLM plugins (e.g., Presidio integration) to automatically redact personal identifiable information (PII), API keys, or database credentials before sending prompts to external APIs.
• Block System Prompt Overrides: Validate user inputs before passing them to LLMs to prevent jailbreaking and indirect prompt injections.

4. Container Resource Allocation & Limits

Local LLM wrappers can cause high CPU/RAM usage. Enforce strict Docker resource limits so AI jobs never starve critical web services:

version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    restart: always
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    ports:
      - "127.0.0.1:4000:4000"
    deploy:
      resources:
        limits:
          cpus: '1.5'
          memory: 2048M