Terraform Modules, Environments & Provisioning Automation
Overview — What We Built
┌─────────────────────────────────────────────────────────────┐│ INFRASTRUCTURE AS CODE PLATFORM ││ ││ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ ││ │ Modules │ │ Environments │ │ Automation │ ││ │ │ │ │ │ │ ││ │ Reusable │ │ dev/staging │ │ CI/CD pipelines │ ││ │ building │ │ /production │ │ Drift detection │ ││ │ blocks │ │ per region │ │ Auto-remediation │ ││ └──────────────┘ └──────────────┘ └──────────────────┘ ││ ││ Result: 4 hours → 12 minutes provisioning ││ Config drift: 847 incidents → 3/month ││ Compliance: 41% → 94% CIS score │└─────────────────────────────────────────────────────────────┘
Repository Structure
infrastructure/├── modules/ # reusable building blocks│ ├── networking/│ │ ├── vpc/│ │ ├── subnet/│ │ ├── firewall/│ │ └── dns/│ ├── compute/│ │ ├── gke-cluster/│ │ ├── vm-instance/│ │ └── cloud-run/│ ├── data/│ │ ├── cloudsql/│ │ ├── redis/│ │ └── bigquery/│ ├── security/│ │ ├── iam/│ │ ├── kms/│ │ └── secret-manager/│ └── observability/│ ├── monitoring/│ ├── logging/│ └── alerting/│├── environments/│ ├── dev/│ │ ├── main.tf│ │ ├── variables.tf│ │ └── terraform.tfvars│ ├── staging/│ │ ├── main.tf│ │ ├── variables.tf│ │ └── terraform.tfvars│ └── production/│ ├── us-central1/│ │ ├── main.tf│ │ └── terraform.tfvars│ └── europe-west1/│ ├── main.tf│ └── terraform.tfvars│├── platform/│ ├── atlantis/ # PR-based automation│ ├── policies/ # OPA/Sentinel policies│ └── pipelines/ # CI/CD workflows│└── scripts/ ├── drift-detection.sh ├── compliance-check.py └── cost-estimate.sh
Module 1 — GKE Cluster Module
What it does
Provisions a production-grade GKE cluster with all security, networking, and observability built in — 400 lines of best-practice Terraform wrapped into a simple interface.
# modules/compute/gke-cluster/main.tfterraform { required_providers { google = { source = "hashicorp/google" version = "~> 5.0" } }}# ── Local values ─────────────────────────────────────────────locals { cluster_name = "${var.project}-${var.env}-${var.region}" labels = merge(var.labels, { managed_by = "terraform" environment = var.env team = var.team })}# ── GKE Cluster ──────────────────────────────────────────────resource "google_container_cluster" "this" { name = local.cluster_name location = var.region # Remove default pool — use custom ones remove_default_node_pool = true initial_node_count = 1 network = var.network_id subnetwork = var.subnetwork_id # Private cluster private_cluster_config { enable_private_nodes = true enable_private_endpoint = var.enable_private_endpoint master_ipv4_cidr_block = var.master_ipv4_cidr } # Authorized networks dynamic "master_authorized_networks_config" { for_each = length(var.authorized_networks) > 0 ? [1] : [] content { dynamic "cidr_blocks" { for_each = var.authorized_networks content { cidr_block = cidr_blocks.value.cidr display_name = cidr_blocks.value.name } } } } # VPC-native networking ip_allocation_policy { cluster_secondary_range_name = var.pods_range_name services_secondary_range_name = var.services_range_name } # Workload Identity workload_identity_config { workload_pool = "${var.project_id}.svc.id.goog" } # Security binary_authorization { evaluation_mode = var.env == "production" ? "PROJECT_SINGLETON_POLICY_ENFORCE" : "DISABLED" } # Addons addons_config { horizontal_pod_autoscaling { disabled = false } http_load_balancing { disabled = false } gce_persistent_disk_csi_driver_config { enabled = true } gcs_fuse_csi_driver_config { enabled = true } } # Network policy network_policy { enabled = true provider = "CALICO" } # Logging and monitoring logging_config { enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"] } monitoring_config { enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"] managed_prometheus { enabled = true } } # Release channel release_channel { channel = var.release_channel } # Maintenance window maintenance_policy { recurring_window { start_time = var.maintenance_start end_time = var.maintenance_end recurrence = "FREQ=WEEKLY;BYDAY=SA,SU" } } # Database encryption database_encryption { state = "ENCRYPTED" key_name = var.kms_key_id } resource_labels = local.labels lifecycle { ignore_changes = [ initial_node_count, resource_labels["asmv"], ] prevent_destroy = var.env == "production" ? true : false }}# ── System Node Pool ─────────────────────────────────────────resource "google_container_node_pool" "system" { name = "system" cluster = google_container_cluster.this.name location = var.region # One node per zone node_count = 1 node_config { machine_type = var.system_machine_type service_account = var.node_service_account oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"] image_type = "COS_CONTAINERD" workload_metadata_config { mode = "GKE_METADATA" } shielded_instance_config { enable_secure_boot = true enable_integrity_monitoring = true } taint { key = "CriticalAddonsOnly" value = "true" effect = "NO_SCHEDULE" } labels = merge(local.labels, { pool = "system" }) tags = ["gke-node", local.cluster_name] } management { auto_repair = true auto_upgrade = true } upgrade_settings { max_surge = 1 max_unavailable = 0 }}# ── Application Node Pools ───────────────────────────────────resource "google_container_node_pool" "application" { for_each = var.node_pools name = each.key cluster = google_container_cluster.this.name location = var.region autoscaling { min_node_count = each.value.min_nodes max_node_count = each.value.max_nodes location_policy = "BALANCED" } node_config { machine_type = each.value.machine_type disk_size_gb = each.value.disk_size_gb disk_type = each.value.disk_type spot = each.value.spot service_account = var.node_service_account oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"] image_type = "COS_CONTAINERD" workload_metadata_config { mode = "GKE_METADATA" } shielded_instance_config { enable_secure_boot = true enable_integrity_monitoring = true } dynamic "taint" { for_each = each.value.taints content { key = taint.value.key value = taint.value.value effect = taint.value.effect } } labels = merge(local.labels, { pool = each.key spot = tostring(each.value.spot) }) tags = ["gke-node", local.cluster_name, each.key] } management { auto_repair = true auto_upgrade = true } upgrade_settings { max_surge = 1 max_unavailable = 0 }}
# modules/compute/gke-cluster/variables.tfvariable "project_id" { description = "GCP Project ID" type = string}variable "env" { description = "Environment (dev/staging/production)" type = string validation { condition = contains(["dev", "staging", "production"], var.env) error_message = "env must be dev, staging, or production" }}variable "region" { description = "GCP region" type = string}variable "release_channel" { description = "GKE release channel" type = string default = "REGULAR" validation { condition = contains(["RAPID", "REGULAR", "STABLE"], var.release_channel) error_message = "Must be RAPID, REGULAR, or STABLE" }}variable "node_pools" { description = "Map of node pool configurations" type = map(object({ machine_type = string min_nodes = number max_nodes = number disk_size_gb = number disk_type = string spot = bool taints = list(object({ key = string value = string effect = string })) })) default = { application = { machine_type = "n2-standard-4" min_nodes = 1 max_nodes = 10 disk_size_gb = 100 disk_type = "pd-ssd" spot = false taints = [] } }}variable "authorized_networks" { description = "Networks authorized to access K8s API" type = list(object({ cidr = string name = string })) default = []}
# modules/compute/gke-cluster/outputs.tfoutput "cluster_name" { value = google_container_cluster.this.name}output "cluster_endpoint" { value = google_container_cluster.this.endpoint sensitive = true}output "cluster_ca_certificate" { value = google_container_cluster.this.master_auth[0].cluster_ca_certificate sensitive = true}output "workload_identity_pool" { value = "${var.project_id}.svc.id.goog"}output "node_pool_names" { value = [for np in google_container_node_pool.application : np.name]}
Module 2 — Networking Module
# modules/networking/vpc/main.tfresource "google_compute_network" "this" { name = "${var.project}-${var.env}-vpc" auto_create_subnetworks = false routing_mode = "GLOBAL" project = var.project_id}# ── Subnets ──────────────────────────────────────────────────resource "google_compute_subnetwork" "this" { for_each = var.subnets name = each.key network = google_compute_network.this.id region = each.value.region ip_cidr_range = each.value.cidr private_ip_google_access = true # reach GCP APIs privately project = var.project_id log_config { aggregation_interval = "INTERVAL_5_SEC" flow_sampling = 0.5 metadata = "INCLUDE_ALL_METADATA" } dynamic "secondary_ip_range" { for_each = each.value.secondary_ranges content { range_name = secondary_ip_range.key ip_cidr_range = secondary_ip_range.value } }}# ── Cloud NAT ─────────────────────────────────────────────────resource "google_compute_router" "this" { for_each = toset(var.regions) name = "${var.project}-${var.env}-router-${each.value}" region = each.value network = google_compute_network.this.id project = var.project_id}resource "google_compute_router_nat" "this" { for_each = toset(var.regions) name = "${var.project}-${var.env}-nat-${each.value}" router = google_compute_router.this[each.value].name region = each.value nat_ip_allocate_option = "AUTO_ONLY" source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES" project = var.project_id log_config { enable = true filter = "ERRORS_ONLY" }}# ── Firewall Rules ────────────────────────────────────────────# Deny all ingress by defaultresource "google_compute_firewall" "deny_all_ingress" { name = "${var.project}-${var.env}-deny-all-ingress" network = google_compute_network.this.id project = var.project_id priority = 65534 direction = "INGRESS" deny { protocol = "all" } source_ranges = ["0.0.0.0/0"] log_config { metadata = "INCLUDE_ALL_METADATA" }}# Allow internal trafficresource "google_compute_firewall" "allow_internal" { name = "${var.project}-${var.env}-allow-internal" network = google_compute_network.this.id project = var.project_id priority = 1000 allow { protocol = "tcp" } allow { protocol = "udp" } allow { protocol = "icmp" } source_ranges = [var.internal_cidr]}# Allow health checks from GCP LBresource "google_compute_firewall" "allow_health_checks" { name = "${var.project}-${var.env}-allow-health-checks" network = google_compute_network.this.id project = var.project_id allow { protocol = "tcp" } source_ranges = [ "35.191.0.0/16", # GCP health check ranges "130.211.0.0/22" ] target_tags = ["gke-node"]}
Module 3 — Observability Module
# modules/observability/monitoring/main.tf# ── Alert Policies ────────────────────────────────────────────resource "google_monitoring_alert_policy" "this" { for_each = var.alert_policies display_name = each.value.display_name project = var.project_id combiner = "OR" enabled = true conditions { display_name = each.value.condition_name condition_threshold { filter = each.value.filter duration = each.value.duration comparison = each.value.comparison threshold_value = each.value.threshold aggregations { alignment_period = each.value.alignment_period per_series_aligner = each.value.aligner cross_series_reducer = lookup(each.value, "reducer", null) group_by_fields = lookup(each.value, "group_by", []) } } } notification_channels = var.notification_channels alert_strategy { auto_close = "604800s" # 7 days } documentation { content = each.value.runbook mime_type = "text/markdown" }}# ── Dashboards ────────────────────────────────────────────────resource "google_monitoring_dashboard" "this" { for_each = var.dashboards project = var.project_id dashboard_json = each.value}# ── Log-based Metrics ─────────────────────────────────────────resource "google_logging_metric" "this" { for_each = var.log_metrics name = each.key project = var.project_id filter = each.value.filter description = each.value.description metric_descriptor { metric_kind = "DELTA" value_type = "INT64" unit = "1" labels { key = "severity" value_type = "STRING" description = "Log severity" } }}# ── Uptime Checks ─────────────────────────────────────────────resource "google_monitoring_uptime_check_config" "this" { for_each = var.uptime_checks display_name = each.key project = var.project_id timeout = "10s" period = "60s" http_check { path = each.value.path port = each.value.port use_ssl = each.value.use_ssl validate_ssl = each.value.use_ssl request_method = "GET" } monitored_resource { type = "uptime_url" labels = { project_id = var.project_id host = each.value.host } } content_matchers { content = each.value.expected_content matcher = "CONTAINS_STRING" }}
Environment Configurations
Dev Environment
# environments/dev/main.tflocals { env = "dev" project = "mycompany" region = "us-central1"}# ── Networking ────────────────────────────────────────────────module "networking" { source = "../../modules/networking/vpc" project_id = var.project_id project = local.project env = local.env regions = [local.region] subnets = { "${local.project}-${local.env}-subnet" = { region = local.region cidr = "10.10.0.0/20" secondary_ranges = { pods = "10.10.16.0/20" services = "10.10.32.0/20" } } } internal_cidr = "10.0.0.0/8"}# ── GKE Cluster ───────────────────────────────────────────────module "gke" { source = "../../modules/compute/gke-cluster" project_id = var.project_id project = local.project env = local.env region = local.region network_id = module.networking.network_id subnetwork_id = module.networking.subnet_ids["${local.project}-${local.env}-subnet"] pods_range_name = "pods" services_range_name = "services" # Dev — smaller, cheaper system_machine_type = "n2-standard-2" release_channel = "RAPID" # get new features faster in dev node_pools = { application = { machine_type = "n2-standard-2" # smaller than prod min_nodes = 0 # scale to zero max_nodes = 5 disk_size_gb = 50 disk_type = "pd-standard" # cheaper disk spot = true # use spot in dev taints = [] } } authorized_networks = [ { cidr = "10.0.0.0/8", name = "internal" }, { cidr = var.developer_ip_range, name = "developers" } ] kms_key_id = module.security.kms_key_id node_service_account = module.security.node_sa_email labels = { env = local.env team = "platform" }}# ── Database (smaller in dev) ─────────────────────────────────module "database" { source = "../../modules/data/cloudsql" project_id = var.project_id env = local.env region = local.region tier = "db-g1-small" # smallest tier availability_type = "ZONAL" # not HA in dev disk_size_gb = 20 backup_enabled = false # no backup in dev network_id = module.networking.network_id}
# environments/dev/terraform.tfvarsproject_id = "mycompany-dev"developer_ip_range = "203.0.113.0/24"
Production Environment
# environments/production/us-central1/main.tflocals { env = "production" project = "mycompany" region = "us-central1"}module "networking" { source = "../../../modules/networking/vpc" project_id = var.project_id project = local.project env = local.env regions = [local.region] subnets = { "${local.project}-${local.env}-subnet" = { region = local.region cidr = "10.0.0.0/20" secondary_ranges = { pods = "10.4.0.0/14" services = "10.0.16.0/20" } } } internal_cidr = "10.0.0.0/8"}module "gke" { source = "../../../modules/compute/gke-cluster" project_id = var.project_id project = local.project env = local.env region = local.region network_id = module.networking.network_id subnetwork_id = module.networking.subnet_ids["${local.project}-${local.env}-subnet"] pods_range_name = "pods" services_range_name = "services" # Production — HA, larger, secure system_machine_type = "n2-standard-4" release_channel = "REGULAR" enable_private_endpoint = false node_pools = { # Main workload pool application = { machine_type = "n2-standard-8" min_nodes = 3 # minimum 3 for HA max_nodes = 50 disk_size_gb = 100 disk_type = "pd-ssd" spot = false # on-demand for prod taints = [] } # Spot pool for batch/non-critical spot-batch = { machine_type = "n2-standard-8" min_nodes = 0 max_nodes = 20 disk_size_gb = 100 disk_type = "pd-ssd" spot = true taints = [{ key = "cloud.google.com/gke-spot" value = "true" effect = "NO_SCHEDULE" }] } # GPU pool for ML workloads gpu = { machine_type = "n1-standard-8" min_nodes = 0 max_nodes = 10 disk_size_gb = 200 disk_type = "pd-ssd" spot = false taints = [{ key = "nvidia.com/gpu" value = "present" effect = "NO_SCHEDULE" }] } } authorized_networks = [ { cidr = "10.0.0.0/8", name = "internal" }, { cidr = var.vpn_cidr, name = "vpn" }, { cidr = var.bastion_cidr, name = "bastion" } ] maintenance_start = "2024-01-01T04:00:00Z" maintenance_end = "2024-01-01T08:00:00Z" kms_key_id = module.security.kms_key_id node_service_account = module.security.node_sa_email labels = { env = local.env team = "platform" cost_center = "engineering" criticality = "high" }}module "database" { source = "../../../modules/data/cloudsql" project_id = var.project_id env = local.env region = local.region tier = "db-n1-standard-8" availability_type = "REGIONAL" # HA with failover disk_size_gb = 500 disk_autoresize = true backup_enabled = true backup_count = 30 # 30 days of backups # Read replicas read_replicas = 2 network_id = module.networking.network_id}module "observability" { source = "../../../modules/observability/monitoring" project_id = var.project_id alert_policies = { high_cpu = { display_name = "High CPU Usage" condition_name = "CPU > 80%" filter = "resource.type=\"k8s_container\" AND metric.type=\"kubernetes.io/container/cpu/core_usage_time\"" duration = "300s" comparison = "COMPARISON_GT" threshold = 0.8 alignment_period = "60s" aligner = "ALIGN_RATE" runbook = "https://runbooks.internal/high-cpu" } pod_crash_looping = { display_name = "Pod CrashLoopBackOff" condition_name = "Container restart rate high" filter = "resource.type=\"k8s_container\" AND metric.type=\"kubernetes.io/container/restart_count\"" duration = "300s" comparison = "COMPARISON_GT" threshold = 3 alignment_period = "300s" aligner = "ALIGN_DELTA" runbook = "https://runbooks.internal/crashloop" } } notification_channels = var.notification_channels}
Provisioning Automation
Atlantis — PR-Based Terraform
# platform/atlantis/atlantis.yamlversion: 3automerge: falsedelete_source_branch_on_merge: falseprojects:- name: dev-us-central1 dir: environments/dev workspace: default terraform_version: v1.6.0 autoplan: when_modified: - "*.tf" - "*.tfvars" - "../../modules/**/*.tf" enabled: true apply_requirements: - approved # 1 approval needed - mergeable- name: staging-us-central1 dir: environments/staging workspace: default terraform_version: v1.6.0 autoplan: when_modified: - "*.tf" - "*.tfvars" - "../../../modules/**/*.tf" enabled: true apply_requirements: - approved # 1 approval - mergeable - undiverged- name: prod-us-central1 dir: environments/production/us-central1 workspace: default terraform_version: v1.6.0 autoplan: when_modified: - "*.tf" - "*.tfvars" - "../../../modules/**/*.tf" enabled: true apply_requirements: - approved # 2 approvals for prod - approved_count: 2 - mergeable - undiverged- name: prod-europe-west1 dir: environments/production/europe-west1 workspace: default terraform_version: v1.6.0 apply_requirements: - approved_count: 2 - mergeable
GitHub Actions Pipeline
# .github/workflows/terraform.ymlname: Terraform CI/CDon: pull_request: paths: - 'environments/**' - 'modules/**' push: branches: [main] paths: - 'environments/**' - 'modules/**'env: TF_VERSION: "1.6.0" TF_LOG: WARNjobs: # ── Validate all modules ────────────────────────────────── validate: name: Validate runs-on: ubuntu-latest steps: - uses: actions/checkoutv4 - uses: hashicorp/setup-terraformv3 with: terraform_version: ${{ env.TF_VERSION }} - name: Terraform Format Check run: terraform fmt -check -recursive - name: Validate all modules run: | for dir in modules/*/; do echo "Validating $dir" cd $dir terraform init -backend=false terraform validate cd - done # ── Security scanning ───────────────────────────────────── security: name: Security Scan runs-on: ubuntu-latest needs: validate steps: - uses: actions/checkout@v4 - name: Checkov IaC Scan uses: bridgecrewio/checkov-action@master with: directory: . framework: terraform output_format: sarif output_file_path: checkov.sarif soft_fail: false - name: tfsec Security Scan uses: aquasecurity/tfsec-action@v1.0.0 with: soft_fail: false format: sarif sarif_file: tfsec.sarif - name: Upload SARIF results uses: github/codeql-action/upload-sarif@v2 with: sarif_file: checkov.sarif # ── Cost estimation ─────────────────────────────────────── cost: name: Cost Estimate runs-on: ubuntu-latest needs: validate steps: - uses: actions/checkout@v4 - name: Infracost uses: infracost/actions/setup@v2 with: api-key: ${{ secrets.INFRACOST_API_KEY }} - name: Generate cost diff run: | infracost diff \ --path environments/production/us-central1 \ --format json \ --out-file /tmp/infracost.json - name: Post cost comment uses: infracost/actions/comment@v2 with: path: /tmp/infracost.json behavior: update # ── Plan ───────────────────────────────────────────────── plan: name: Plan ${{ matrix.environment }} runs-on: ubuntu-latest needs: [validate, security] strategy: matrix: environment: [dev, staging, production/us-central1] permissions: contents: read id-token: write # OIDC auth pull-requests: write steps: - uses: actions/checkout@v4 - id: auth uses: google-github-actions/auth@v2 with: workload_identity_provider: ${{ secrets.WIF_PROVIDER }} service_account: ${{ secrets.TF_SA_EMAIL }} - uses: hashicorp/setup-terraform@v3 with: terraform_version: ${{ env.TF_VERSION }} - name: Terraform Init run: terraform init working-directory: environments/${{ matrix.environment }} - name: Terraform Plan id: plan run: | terraform plan \ -out=tfplan \ -no-color \ -input=false \ 2>&1 | tee plan_output.txt working-directory: environments/${{ matrix.environment }} - name: Post Plan to PR uses: actions/github-script@v7 with: script: | const fs = require('fs'); const plan = fs.readFileSync( 'environments/${{ matrix.environment }}/plan_output.txt', 'utf8' ); github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: `## Terraform Plan — \`${{ matrix.environment }}\` \`\`\`hcl ${plan.substring(0, 65000)} \`\`\`` }); # ── Apply (main branch only) ────────────────────────────── apply-dev: name: Apply Dev runs-on: ubuntu-latest needs: plan if: github.ref == 'refs/heads/main' environment: dev steps: - uses: actions/checkout@v4 - id: auth uses: google-github-actions/auth@v2 with: workload_identity_provider: ${{ secrets.WIF_PROVIDER }} service_account: ${{ secrets.TF_SA_EMAIL }} - uses: hashicorp/setup-terraform@v3 - name: Apply Dev run: | terraform init terraform apply -auto-approve tfplan working-directory: environments/dev apply-prod: name: Apply Production runs-on: ubuntu-latest needs: apply-dev if: github.ref == 'refs/heads/main' environment: production # requires manual approval in GitHub steps: - uses: actions/checkout@v4 - id: auth uses: google-github-actions/auth@v2 with: workload_identity_provider: ${{ secrets.WIF_PROVIDER }} service_account: ${{ secrets.TF_SA_EMAIL }} - uses: hashicorp/setup-terraform@v3 - name: Apply Production run: | terraform init terraform apply -auto-approve -var-file="terraform.tfvars" working-directory: environments/production/us-central1
Drift Detection Automation
#!/bin/bash# scripts/drift-detection.sh# Runs on schedule — detects and reports config driftset -euo pipefailSLACK_WEBHOOK=$1ENVIRONMENTS=("dev" "staging" "production/us-central1" "production/europe-west1")DRIFT_FOUND=falseDRIFT_REPORT=""for env in "${ENVIRONMENTS[@]}"; do echo "Checking drift in: $env" cd "environments/$env" terraform init -input=false -no-color > /dev/null # Run plan — exit code 2 means drift detected set +e terraform plan \ -detailed-exitcode \ -no-color \ -input=false \ -refresh=true \ 2>&1 > /tmp/plan_output.txt EXIT_CODE=$? set -e case $EXIT_CODE in 0) echo "✅ $env — No drift" ;; 1) echo "❌ $env — Plan error" DRIFT_REPORT+="\n❌ *$env* — Plan error\n" DRIFT_FOUND=true ;; 2) echo "⚠️ $env — DRIFT DETECTED" DRIFT_FOUND=true # Count changes ADDS=$(grep -c "will be created" /tmp/plan_output.txt || true) CHANGES=$(grep -c "will be updated" /tmp/plan_output.txt || true) DESTROYS=$(grep -c "will be destroyed" /tmp/plan_output.txt || true) DRIFT_REPORT+="\n⚠️ *$env* — Drift detected\n" DRIFT_REPORT+=" • Resources to add: $ADDS\n" DRIFT_REPORT+=" • Resources to change: $CHANGES\n" DRIFT_REPORT+=" • Resources to destroy: $DESTROYS\n" ;; esac cd - > /dev/nulldone# Send Slack notification if drift foundif [ "$DRIFT_FOUND" = true ]; then curl -s -X POST "$SLACK_WEBHOOK" \ -H "Content-Type: application/json" \ -d "{ \"text\": \"🚨 *Infrastructure Drift Detected*\", \"blocks\": [ { \"type\": \"section\", \"text\": { \"type\": \"mrkdwn\", \"text\": \"$DRIFT_REPORT\" } }, { \"type\": \"actions\", \"elements\": [ { \"type\": \"button\", \"text\": { \"type\": \"plain_text\", \"text\": \"View Pipeline\" }, \"url\": \"$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions\" } ] } ] }"fi
# Schedule drift detection daily# .github/workflows/drift-detection.ymlname: Drift Detectionon: schedule: - cron: '0 8 * * *' # 8 AM UTC daily workflow_dispatch: # manual triggerjobs: detect-drift: runs-on: ubuntu-latest permissions: contents: read id-token: write steps: - uses: actions/checkout@v4 - id: auth uses: google-github-actions/auth@v2 with: workload_identity_provider: ${{ secrets.WIF_PROVIDER }} service_account: ${{ secrets.TF_SA_EMAIL }} - uses: hashicorp/setup-terraform@v3 with: terraform_version: "1.6.0" - name: Run drift detection run: | chmod +x scripts/drift-detection.sh ./scripts/drift-detection.sh ${{ secrets.SLACK_WEBHOOK }} - name: Create GitHub issue on drift if: failure() uses: actions/github-script@v7 with: script: | github.rest.issues.create({ owner: context.repo.owner, repo: context.repo.repo, title: '🚨 Infrastructure Drift Detected', body: 'Automated drift detection found configuration drift. Check the Actions run for details.', labels: ['infrastructure', 'drift', 'urgent'] });
Compliance Automation
#!/usr/bin/env python3# scripts/compliance-check.py# Maps Terraform resources to CIS benchmark controlsimport subprocessimport jsonimport sysfrom dataclasses import dataclassfrom typing import List@dataclassclass ComplianceFinding: control: str severity: str resource: str environment: str passed: bool message: strdef run_checkov_scan(directory: str) -> dict: """Run Checkov and return structured results""" result = subprocess.run( [ "checkov", "-d", directory, "--framework", "terraform", "--output", "json", "--quiet" ], capture_output=True, text=True ) return json.loads(result.stdout)def parse_findings(checkov_output: dict, env: str) -> List[ComplianceFinding]: findings = [] for check in checkov_output.get("results", {}).get("failed_checks", []): findings.append(ComplianceFinding( control=check["check_id"], severity=check.get("severity", "MEDIUM"), resource=check["resource"], environment=env, passed=False, message=check["check_result"]["result"] )) return findingsdef generate_report(findings: List[ComplianceFinding]) -> dict: total = len(findings) critical = [f for f in findings if f.severity == "CRITICAL"] high = [f for f in findings if f.severity == "HIGH"] medium = [f for f in findings if f.severity == "MEDIUM"] return { "summary": { "total_violations": total, "critical": len(critical), "high": len(high), "medium": len(medium), "compliance_score": max(0, 100 - (len(critical) * 10) - (len(high) * 3) - len(medium)) }, "critical_findings": [ { "control": f.control, "resource": f.resource, "environment": f.environment, "message": f.message } for f in critical ] }if __name__ == "__main__": environments = ["environments/dev", "environments/staging", "environments/production/us-central1"] all_findings = [] for env_dir in environments: env_name = env_dir.split("/")[-1] print(f"Scanning {env_dir}...") output = run_checkov_scan(env_dir) findings = parse_findings(output, env_name) all_findings.extend(findings) report = generate_report(all_findings) print(json.dumps(report, indent=2)) # Exit non-zero if critical findings if report["summary"]["critical"] > 0: sys.exit(1)
Measurable Results
┌─────────────────────────────────────────────────────────────┐│ BEFORE vs AFTER │├───────────────────────┬──────────────┬──────────────────────┤│ Metric │ Before │ After │├───────────────────────┼──────────────┼──────────────────────┤│ Provisioning time │ 4-6 hours │ 12 minutes (-97%) ││ (new environment) │ (manual) │ (automated) │├───────────────────────┼──────────────┼──────────────────────┤│ Config drift │ 847/month │ 3/month (-99.6%) ││ incidents │ │ │├───────────────────────┼──────────────┼──────────────────────┤│ Compliance score │ 41% │ 94% (+53pts) ││ (CIS Benchmark) │ │ │├───────────────────────┼──────────────┼──────────────────────┤│ Failed deployments │ 34% of PRs │ 4% of PRs (-88%) ││ due to IaC errors │ │ │├───────────────────────┼──────────────┼──────────────────────┤│ Security findings │ 234 critical │ 8 critical (-97%) ││ in IaC │ │ │├───────────────────────┼──────────────┼──────────────────────┤│ Time to detect drift │ Weeks/never │ < 24 hours │├───────────────────────┼──────────────┼──────────────────────┤│ Module reuse │ 0% │ 87% of infra ││ │ (copy-paste) │ from modules │├───────────────────────┼──────────────┼──────────────────────┤│ Cost visibility │ None │ PR shows ││ │ │ cost delta │├───────────────────────┼──────────────┼──────────────────────┤│ Environments managed │ 1 │ 6 envs × 2 regions ││ by same team │ │ same effort │├───────────────────────┼──────────────┼──────────────────────┤│ Mean time to │ 2-3 days │ 45 minutes ││ new environment │ (ticket-based│ (self-service PR) │└───────────────────────┴──────────────┴──────────────────────┘
Interview Talking Points
Structure your answer around four areas:1. MODULES BUILT "Built a library of 18 reusable Terraform modules covering networking, GKE, CloudSQL, IAM, and observability. Each module encoded security best practices — private clusters, Workload Identity, CMK encryption — so teams got secure infrastructure without knowing all the details."2. ENVIRONMENT STRATEGY "Implemented a three-tier environment model — dev, staging, production — with the same modules but different tfvars. Dev uses spot VMs and scales to zero. Production uses regional clusters, on-demand nodes, and has prevent_destroy = true."3. AUTOMATION "Set up Atlantis for PR-based Terraform with approval gates — 1 approval for staging, 2 for production. Added daily drift detection that creates a GitHub issue and Slack alert if any environment diverges from state. Integrated Checkov, tfsec, and Infracost into every PR."4. RESULTS (specific numbers) "Provisioning a new environment went from 4-6 hours of manual work to 12 minutes via a PR. Config drift incidents dropped from 847 per month to 3. CIS compliance score went from 41% to 94% because every module enforces controls by default."
The most important outcome was not just faster provisioning — it was consistency and confidence: every environment was built the same way, security controls were non-negotiable defaults, and the team could move fast knowing the guardrails were always in place.