a secure, production-ready docker-compose.yml for LiteLLM

techhadoop ai, security

Here is a secure, production-ready docker-compose.yml for LiteLLM that implements local network binding, explicit CPU/memory constraints, non-root user execution, read-only container filesystems, and strict environment variable handling.

1. Project Directory Structure

Set up your project directory layout as follows before starting the container:

Plaintext

litellm-project/
├── .env                  # Private secrets (DO NOT COMMIT)
├── config.yaml           # LiteLLM routing & proxy settings
└── docker-compose.yml    # Hardened container stack

2. File Configurations

.env (Secrets File)

Create a .env file in the same directory. This keeps sensitive API keys isolated from your repository code.

Ini, TOML

# Upstream LLM API Keys
OPENAI_API_KEY=sk-proj-your-actual-openai-api-key-here
ANTHROPIC_API_KEY=sk-ant-your-actual-anthropic-api-key-here
# LiteLLM Administration Key (Used for generating virtual keys and proxy admin)
LITELLM_MASTER_KEY=sk-master-litellm-secret-key-change-this

config.yaml (LiteLLM Model & Security Config)

Define your backend models and security settings:

YAML

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  # Enable database / budget features if using LiteLLM database mode

docker-compose.yml (Hardened Deployment)

YAML

version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-v1.59.0
    container_name: litellm-proxy
    restart: unless-stopped
    
    # 1. Local Network Binding (Prevents exposure on public/LAN interfaces)
    ports:
      - "127.0.0.1:4000:4000"
    # 2. Command Execution with Config Mapping
    command: [ "--config", "/app/config.yaml", "--port", "4000", "--num_workers", "2" ]
    # 3. Mount Config & Read-Only Temp Volume
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - tmp_data:/tmp
    # 4. Environment Variables loaded from .env
    env_file:
      - .env
    # 5. Security & Isolation Enhancements
    read_only: true               # Prevents malicious writes to the container filesystem
    user: "1000:1000"             # Runs as a non-root user
    security_opt:
      - no-new-privileges:true    # Blocks privilege escalation attacks
    cap_drop:
      - ALL                       # Drops all Linux kernel capabilities
    # 6. Resource Limits (Prevents container from starving host CPU/RAM)
    deploy:
      resources:
        limits:
          cpus: '1.5'
          memory: 1536M
        reservations:
          cpus: '0.25'
          memory: 256M
    # 7. Health Check
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 15s
volumes:
  tmp_data:
    driver: local

Step-by-Step Deployment Instructions

1.Set Secure File Permissions for Secrets:

Ensure the .env file is accessible only by your host user:

chmod 600 .env

Verification: Run ls -l .env and confirm the permission bit shows -rw-------.

2.Start the LiteLLM Proxy Stack:

Launch the container in detached mode:

docker compose up -d

Verification: Run docker compose ps to verify the container state is Up (healthy).

3.Test Local Binding Security:

Verify that the proxy endpoints respond on 127.0.0.1 using your configured master key:

Bash

curl -i http://127.0.0.1:4000/health/readiness \
  -H "Authorization: Bearer sk-master-litellm-secret-key-change-this"

Verification: Expect an HTTP/1.1 200 OK response with {"status": "connected"}.

Leave a Reply