In may setup, “retention” actually applies to two different things: Prometheus metrics (the graphs) and Ubuntu system logs (the text files).

By default, they have very different lifespans.

1. Prometheus Metrics (The Graphs)

If you didn’t specify a retention time when you ran your Prometheus Docker container, it uses the default.

• Default Duration: 15 days.
• What happens after? Prometheus uses a “First-In-First-Out” system. Once data hits day 16, the data from day 1 is deleted to make room.
• How to change it: If you want 30 days of history to show your Director month-over-month trends, you need to add this flag to your docker run command for Prometheus:--storage.tsdb.retention.time=30d

2. Ubuntu System Logs (/var/log)

This is handled by a service called logrotate. It manages things like your mail.log, syslog, and auth.log.

• Default Duration: Usually 4 weeks (28 days).
• How it works: It keeps 4 “rotated” files. Every Sunday, it compresses the current log and deletes the oldest one.
• How to check your specific settings:Bashcat /etc/logrotate.d/rsyslog Look for the number next to rotate. If it says 4, and the interval is weekly, you have 28 days.

3. Docker Container Logs

This is the danger zone. By default, Docker container logs (like the ones for cadvisor or node-exporter) have no limit.

If a container starts throwing thousands of errors, the log file will grow until it fills your entire hard drive. Since we are doing a Pilot Group, you should verify your Docker logging driver.

The “Safe” way to run your containers:

Add these flags to your docker run commands to ensure you only keep 3 files of 10MB each:

Bash

--log-opt max-size=10m --log-opt max-file=3

Summary Table

Recommendation for your 20 Servers

For an Executive Director’s report, 15 days is usually too short. Most admins prefer 90 days for Prometheus so they can compare “This Quarter vs. Last Quarter.”

To check how much disk space your current Prometheus data is taking:

Bash

du -sh /var/lib/docker/volumes/<prometheus_volume_name>/_data

To update your Prometheus retention to 90 days, you need to restart the container with a specific flag. Since you are likely running this via a docker run command, we will stop the old one and start the new one with the updated storage policy.

1. Update Prometheus to 90-Day Retention

Run these commands on your Central Monitoring Server:

Bash

# 1. Stop and remove the existing Prometheus container
docker stop prometheus
docker rm prometheus
# 2. Start it again with the 90-day retention flag
docker run -d \
  --name=prometheus \
  --restart=always \
  --publish=9090:9090 \
  -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v prometheus-data:/prometheus \
  prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.retention.time=90d

2. Why 90 Days? (The Strategy)

Setting it to 90 days is the “sweet spot” for several reasons:

• Quarterly Reporting: You can show your Executive Director performance trends across a full 3-month business quarter.
• Capacity Planning: 90 days of data allows the Predictive AI (the predict_linear we discussed) to be much more accurate because it has a longer history to analyze.
• Storage Management: Prometheus is very efficient. For 20 servers, 90 days of metrics will likely only consume about 5GB to 10GB of disk space.

3. Verify the Change

Once the container is back up, you can verify that the new setting is active:

1. Open your browser to http://<CENTRAL_IP>:9090/status.
2. Look for Command-Line Flags.
3. Confirm that --storage.tsdb.retention.time=90d is listed there.

4. Pro-Tip: Disk Space Alert

Since you are now keeping 6 times more data than the default, you should add a simple alert in Grafana to monitor the Central Server’s own disk space.

The “Golden Rule” of Monitoring: The monitoring server must be the most stable server in the fleet. If its disk fills up because of long retention, you lose visibility into all other 20 servers.

Next Steps for your Project

Now that your data is safe for the long term:

1. Check the logs: docker logs prometheus to ensure there are no “permission denied” errors on the data folder.
2. Snapshot Check: Since you are modifying the central server, it wouldn’t hurt to take a quick vSphere snapshot of this one too, just like you did for the pilot nodes.