Implementing an ELK Stack (Elasticsearch, Logstash, Kibana) gives you a highly capable platform for log aggregation and analysis. For architecture managing ~20 Linux/Docker servers, you can tie this cleanly into your existing workflow.

A modern production-grade architecture does not typically use Logstash on all 20 nodes because it has a heavy memory footprint. Instead, you deploy lightweight Filebeat agents on your 20 servers to stream data to a central ELK hub.

Part 1: Define the Central ELK Stack

On your central monitoring host, you can deploy the full ELK stack using Portainer or a direct docker-compose.yml file.

1. Configure Host Virtual Memory

Elasticsearch requires memory mapping limits higher than default Linux settings. Before starting the containers, run this on your central host:

sudo sysctl -w vm.max_map_count=262144

To make this setting permanent across server reboots, append vm.max_map_count=262144 to /etc/sysctl.conf.

2. Central docker-compose.yml

Create a directory /opt/elk and save the following file:

YAML

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: central-elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=SuperSecurePassword123 # Choose a strong password
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g" # Allocates 2GB RAM; adjust to host sizing
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    restart: unless-stopped
  logstash:
    image: docker.elastic.co/logstash/logstash:8.15.0
    container_name: central-logstash
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    ports:
      - "5044:5044" # Filebeat connection port
    environment:
      - "LS_JAVA_OPTS=-Xms1g -Xmx1g"
    depends_on:
      - elasticsearch
    restart: unless-stopped
  kibana:
    image: docker.elastic.co/kibana/kibana:8.15.0
    container_name: central-kibana
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=SuperSecurePassword123
    depends_on:
      - elasticsearch
    restart: unless-stopped
volumes:
  es-data:

3. Logstash Pipeline Configuration (logstash.conf)

In the same folder (/opt/elk), create logstash.conf. This instructs Logstash to listen for incoming logs from your 20 servers on port 5044 and parse them into Elasticsearch:

Ruby

input {
  beats {
    port => 5044
  }
}
filter {
  if [container][image][name] =~ /nginx/ {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  # Add other parsing rules/grok filters for system logs here
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "SuperSecurePassword123"
  }
}

Deploy this stack using docker compose up -d or your Portainer Stacks editor.

Part 2: Deploy Filebeat to the 20 Hosts via Ansible

With the central hub running, you need to configure your 20 remote nodes to gather system logs and Docker logs, sending them over the network to Logstash.

1. Filebeat Template File (filebeat.yml.j2)

On your Ansible controller node, create a Jinja2 template file:

YAML

filebeat.inputs:
# Input 1: System syslog and auth logs
- type: log
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log
# Input 2: Dynamic Docker container logs
- type: container
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  processors:
    - add_docker_metadata: ~ # Enriches logs with container names, images, etc.
output.logstash:
  hosts: ["{{ central_logstash_ip }}:5044"]

2. The Ansible Automation Playbook

Create a playbook named deploy-filebeat.yml to automatically install Filebeat across your entire fleet as a native service:

YAML

---
- name: Deploy Filebeat Log Collector
  hosts: docker_servers
  become: true
  vars:
    central_logstash_ip: "192.168.1.10" # Replace with your ELK master server IP
  tasks:
    - name: Download Filebeat Debian package
      ansible.builtin.get_url:
        url: "https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.15.0-amd64.deb"
        dest: "/tmp/filebeat.deb"
        mode: '0644'
    - name: Install Filebeat package
      ansible.builtin.apt:
        deb: "/tmp/filebeat.deb"
        state: present
    - name: Configure Filebeat via Jinja2 Template
      ansible.builtin.template:
        src: filebeat.yml.j2
        dest: /etc/filebeat/filebeat.yml
        owner: root
        group: root
        mode: '0600'
      register: filebeat_config
    - name: Enable and Restart Filebeat Service
      ansible.builtin.systemd:
        name: filebeat
        state: restarted
        enabled: true
        daemon_reload: true
      when: filebeat_config.changed

Execute your playbook to push the log forwarding infrastructure across your servers:

Bash

ansible-playbook -i inventory.ini deploy-filebeat.yml

Part 3: View the Logs in Kibana

1. Open your web browser and navigate to your Kibana portal at http://<central-server-ip>:5601.
2. Log in using the username elastic and the password you defined in your Docker Compose file (SuperSecurePassword123).
3. Navigate to Management -> Stack Management -> Data Views (sometimes labeled Index Patterns).
4. Click Create Data View and define the pattern name as logstash-*. Select @timestamp as your timestamp field.
5. Go to the main menu and click Discover.

You will now see a centralized stream of logs rolling in from your 20 Linux hosts. Because of the add_docker_metadata processor, you can filter logs by specific attributes like container.name, host.name, or container.image.name across your entire network.

Scaling from 20 to 50 Linux servers is a major milestone. While the core architecture pattern (Lightweight Agents $\rightarrow$ Central Queue/Buffer $\rightarrow$ Processing $\rightarrow$ Storage) remains identical, the exact deployment strategy you used for 20 servers will bottle-neck and crash if applied directly to 50 servers.

At 50 servers, you are no longer just handling standard logs; you are running a high-throughput data streaming operation. If 5 servers suddenly experience an app error loop at the same time, they will flood Logstash, exhaust its memory, and cause it to drop logs or crash.

Here is how you must adapt your ELK strategy to safely handle 50 servers.

1. The Architectural Shift: Add a Buffer (Kafka or Redis)

For 20 servers, Filebeat can talk directly to Logstash. For 50 servers, you need to introduce an architecture that includes a Message Queue (like Apache Kafka or Redis) between Filebeat and Logstash.

Why this is mandatory at scale:

• Spike Protection: If your servers generate a sudden burst of millions of log lines, Kafka absorbs the shock. It acts as a shock absorber, writing the logs safely to a temporary disk queue.
• Decoupling: Logstash can now pull logs out of Kafka at its own comfortable pace without getting overwhelmed and crashing.

2. Scale up Elasticsearch Hardware (JVM Heap & Storage)

A single-node Elasticsearch instance running on standard settings will choke on 50 servers. You must adjust your resources:

• Dedicated Production Server: Your ELK Master should be a dedicated machine with at least 32 GB of RAM and fast SSD storage.
• Adjust JVM Heap Size: Update your docker-compose.yml environment variables to give Elasticsearch more memory. A good rule of thumb is giving it 50% of your total system memory, up to 31 GB:YAML- "ES_JAVA_OPTS=-Xms16g -Xmx16g" # Boosted to 16GB for 50 hosts

3. Implement an Index Lifecycle Management (ILM) Policy

50 servers will easily generate 5 GB to 15 GB of raw log data every single day. If you store this indefinitely, Elasticsearch will run out of memory tracking the data indexes.

You must configure a rolling policy in Kibana (Stack Management -> Index Lifecycle Policies) to automatically manage this data lifecycle:

4. Optimize Your Ansible Filebeat Deployments

Your Ansible strategy is still the perfect way to manage 50 hosts, but you need to optimize how Filebeat handles resource boundaries on the target nodes so it doesn’t consume host CPU.

Update your filebeat.yml.j2 template with these enterprise performance tweaks:

YAML

filebeat.inputs:
- type: container
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  # Performance Tuning for 50+ Hosts:
  backoff: 1s                  # How long to wait before checking a file again after reaching EOF
  max_backoff: 10s
  harvester_buffer_size: 16384 # 16KB memory buffer per log file
  queue.mem:
    events: 4096               # Buffer logs in local memory before shipping over the network
    flush.min_events: 512
    flush.timeout: 5s
# If you implemented a queue, point your hosts to Kafka instead of Logstash
output.kafka:
  hosts: ["192.168.1.15:9092"]
  topic: 'docker-logs'
  partition.round_robin:
    reachable_only: false
  required_acks: 1

Summary Checklist for Scaling to 50 Hosts

1. Do not use a single-node setup without bumping the RAM heap size to at least 16 GB.
2. Deploy Redis or Kafka as a buffer if your logs are business-critical and cannot tolerate dropped lines during traffic spikes.
3. Automate Index Deletion from day one so your central cluster doesn’t experience a storage failure in month two.