GKE + Terraform Interview Explanation
How to Frame Your Answer
Interview tip — always structure answers as: 1. WHY — what problem does it solve 2. WHAT — what you built 3. HOW — how it works 4. RESULT — what it achievedNever just list technologies.Say: "I built X to solve Y, which resulted in Z"
Start with the Big Picture
What you say:
“I built a production-grade GKE infrastructure using Terraform — fully automated, modular, and deployed across dev, staging, and production environments. The goal was to eliminate manual cloud provisioning, enforce security by default, and let developers get a new environment in minutes rather than days.”
What we built:┌─────────────────────────────────────────────────────────────┐│ ││ Developer opens PR ││ ↓ ││ GitHub Actions triggers automatically ││ ↓ ││ ┌─────────────────────────────────────────────┐ ││ │ Validate → Security Scan → Plan → Apply │ ││ └─────────────────────────────────────────────┘ ││ ↓ ││ GKE cluster fully provisioned in 12 minutes ││ ↓ ││ Private cluster, Workload Identity, ││ KMS encryption, auto-scaling — all by default ││ │└─────────────────────────────────────────────────────────────┘
Explain the Module Structure
What you say:
“Instead of writing one massive Terraform file, I broke it into three focused modules — networking, security, and GKE. Each module has one responsibility and can be tested independently. This meant the platform team owned the modules, and application teams consumed them without needing to know the security details.”
WHY modules matter:Without modules: With modules:───────────────── ─────────────1 giant main.tf networking/ → VPC, subnets, NAT2000 lines of code security/ → IAM, KMS, SAHard to test gke/ → cluster, node poolsHard to reuseCopy-paste between envs App team just does:Security gets missed module "gke" { source = "./modules/gke" env = "production" } — all security built in
Explain Networking — Why Private Cluster
What you say:
“The cluster is private — nodes have no public IP addresses. All outbound traffic goes through Cloud NAT, and the Kubernetes API is only accessible from authorized networks like our VPN and bastion host. This massively reduces the attack surface.”
Public cluster (avoid):Internet → anyone can reach K8s API → nodes have public IPs → brute force, CVE exploitation riskPrivate cluster (what we built):Internet │ └──▶ Authorized Networks only (office VPN, bastion) │ ▼ K8s API (172.16.0.0/28 — internal only) │ ▼ Worker Nodes (no public IPs) │ ▼ Cloud NAT → outbound to internet (pull images, call APIs)
# What makes it private — explain these two linesprivate_cluster_config { enable_private_nodes = true # nodes get no public IP enable_private_endpoint = false # API accessible via authorized nets master_ipv4_cidr_block = "172.16.0.0/28"}master_authorized_networks_config { cidr_blocks { cidr_block = "10.0.0.0/8" # internal traffic display_name = "internal" } cidr_blocks { cidr_block = "203.0.113.0/24" # office VPN display_name = "office-vpn" }}
Explain VPC-Native Networking
What you say:
“We use VPC-native networking with secondary IP ranges — one range for pods and one for services. This means pods get real VPC IP addresses, so there’s no NAT between pods and GCP services like Cloud SQL or Pub/Sub. It also enables better network policies and visibility.”
VPC-Native (alias IPs):Subnet: 10.0.0.0/20 ← nodes live herePods: 10.4.0.0/14 ← pods get IPs from hereServices: 10.0.16.0/20 ← ClusterIP servicesWhy it matters:├── Pods are first-class VPC citizens├── Firewall rules apply directly to pods├── No double-NAT overhead├── Cloud SQL can whitelist pod IPs directly└── VPC flow logs show pod-level traffic
Explain Security — Workload Identity
What you say:
“The biggest security win was Workload Identity. Before, teams would create service account key files, store them in Kubernetes secrets, and rotate them manually — which is risky and error-prone. With Workload Identity, pods automatically get a GCP identity without any key files. The binding is cryptographic and managed by Google.”
WITHOUT Workload Identity (bad):─────────────────────────────────1. Create GCP service account key file (JSON)2. Store key in K8s secret3. Mount secret into pod4. App reads key file from disk5. Remember to rotate every 90 days6. Key can be stolen from secret storeRisk: leaked key = compromised GCP accountWITH Workload Identity (what we built):────────────────────────────────────────1. K8s Service Account (KSA) created2. KSA annotated with GCP SA email3. Pod uses KSA4. GCP metadata server issues short-lived token5. No key files anywhere6. Token auto-rotates every hourRisk: nothing to steal
# How it works in Terraform# GKE cluster has Workload Identity enabledworkload_identity_config { workload_pool = "${var.project_id}.svc.id.goog"}# Node pool uses GKE_METADATA modeworkload_metadata_config { mode = "GKE_METADATA" # intercepts metadata requests}# GCP SA allows K8s SA to impersonate itresource "google_service_account_iam_member" "wi_binding" { service_account_id = google_service_account.app.name role = "roles/iam.workloadIdentityUser" member = "serviceAccount:myproject.svc.id.goog[production/app-ksa]" # ↑ This K8s SA in this namespace gets this GCP identity}
Explain Security — KMS Encryption
What you say:
“All Kubernetes secrets stored in etcd are encrypted using a customer-managed key in Cloud KMS. This means even if someone gained access to the etcd data directly, they couldn’t read the secrets without also having access to the KMS key — and we control that separately with different IAM permissions.”
Without KMS:etcd stores K8s secrets → base64 encoded onlyAnyone with etcd access reads secrets in plain textWith KMS (what we built):K8s Secret created ↓K8s API Server encrypts it with KMS key ↓Encrypted blob stored in etcd ↓Even raw etcd access shows encrypted dataKMS key is:├── Separate from cluster IAM├── Rotated every 90 days automatically├── Audited — every decrypt operation logged└── prevent_destroy = true — can't accidentally delete
Explain Node Pools Strategy
What you say:
“We have three node pools serving different purposes. The system pool runs cluster components like monitoring and ingress controllers — it’s tainted so application pods don’t land there. The application pool runs business workloads on stable on-demand machines. The spot pool handles batch jobs and can scale to zero — this alone saved about 40% on compute costs.”
Node Pool Strategy:┌──────────────────────────────────────────────────────┐│ SYSTEM POOL ││ n2-standard-2, on-demand, taint: CriticalAddonsOnly ││ Runs: Prometheus, Ingress, Cert-manager, etc. ││ → Isolated from app workloads │└──────────────────────────────────────────────────────┘┌──────────────────────────────────────────────────────┐│ APPLICATION POOL ││ n2-standard-8, on-demand, min=3 max=50 ││ Runs: Production APIs, databases, services ││ → Stable, always available, no eviction │└──────────────────────────────────────────────────────┘┌──────────────────────────────────────────────────────┐│ SPOT POOL ││ n2-standard-4, spot VMs, min=0 max=20 ││ Runs: batch jobs, ML training, data processing ││ → 60-90% cheaper, can be preempted ││ → Tainted — only tolerating pods land here │└──────────────────────────────────────────────────────┘Cost impact:Before: all on-demand → $8,000/monthAfter: mixed pools → $4,800/month (-40%)
# Interviewer might ask: how does the taint work?# Node taint — repels podstaint { key = "cloud.google.com/gke-spot" value = "true" effect = "NO_SCHEDULE" # pods won't land here unless...}# Pod toleration — allows landing on spot nodestolerations:- key: "cloud.google.com/gke-spot" operator: Equal value: "true" effect: NoSchedule # ...they explicitly tolerate it
Explain Autoscaling
What you say:
“We have two levels of autoscaling. Horizontal Pod Autoscaler scales pods when CPU or memory is high. Cluster Autoscaler scales nodes when pods can’t be scheduled because there’s no capacity. Together they handle traffic spikes automatically and scale down during quiet periods to save cost.”
Two-level autoscaling:Traffic spike hits: ↓HPA detects high CPU on pods ↓HPA adds more pods ↓No nodes available for new pods ↓Cluster Autoscaler detects pending pods ↓Cluster Autoscaler adds new node ↓Pods schedule on new node ↓Traffic returns to normal ↓HPA removes excess pods ↓Cluster Autoscaler removes empty node (after 10 min)Result: zero manual intervention cost scales with actual usage
Explain the CI/CD Pipeline
What you say:
“The deployment pipeline has four stages — validate, security scan, plan, apply. Every PR triggers a plan so the team can see exactly what will change before merging. Security scanning with Checkov blocks the pipeline if it finds critical misconfigurations. Production requires a manual approval gate — two senior engineers must approve before the apply runs.”
PR opened: ↓┌─────────────────────────────────────────────────────┐│ 1. VALIDATE (30 seconds) ││ terraform fmt -check ││ terraform validate ││ Fails immediately on syntax errors │└──────────────────────┬──────────────────────────────┘ ↓┌─────────────────────────────────────────────────────┐│ 2. SECURITY SCAN (2 minutes) ││ Checkov — 2000+ IaC checks ││ tfsec — Terraform security scanner ││ Fails on CRITICAL/HIGH findings ││ Blocks deploy if public storage, open ports, etc.│└──────────────────────┬──────────────────────────────┘ ↓┌─────────────────────────────────────────────────────┐│ 3. PLAN (3 minutes) ││ terraform plan for each environment ││ Posts plan output as PR comment ││ Team reviews: what will change? │└──────────────────────┬──────────────────────────────┘ ↓ (PR merged to main)┌─────────────────────────────────────────────────────┐│ 4. APPLY ││ Dev → auto-apply ││ Staging → 1 approval required ││ Production → 2 approvals + manual gate │└─────────────────────────────────────────────────────┘
Explain OIDC Authentication
What you say:
“The pipeline uses OIDC — no GCP credentials stored in GitHub secrets. GitHub proves its identity to GCP using a short-lived cryptographic token. GCP validates the token against our Workload Identity Federation configuration, checks it’s coming from our specific repo and branch, then issues a short-lived access token for that job only.”
Traditional (risky): Store GCP service account key in GitHub secret → Long-lived key, could be leaked → Must rotate manually → If leaked, attacker has permanent accessOIDC (what we use): GitHub job starts ↓ GitHub OIDC provider issues signed JWT ↓ JWT contains: repo, branch, workflow, actor ↓ Terraform job sends JWT to GCP ↓ GCP validates: trusted issuer? correct repo? ↓ GCP issues 1-hour access token ↓ Terraform uses token to create/update resources ↓ Token expires when job endsZero credentials stored anywhereZero rotation neededZero attack surface if GitHub account compromised
Explain State Management
What you say:
“Terraform state is stored in a GCS bucket with versioning enabled. This means the state file is shared across the team — anyone can run Terraform and they’re working with the same view of the world. Versioning acts as a backup — if something goes wrong we can restore a previous state version.”
Why remote state matters:Local state (bad): Alice runs terraform apply → state on Alice's laptop Bob runs terraform apply → state on Bob's laptop → Two different views of reality → Duplicate resources, conflictsRemote state in GCS (what we built): Alice runs terraform apply → reads/writes GCS Bob runs terraform apply → reads same GCS → Single source of truth → State locking prevents simultaneous applies → Versioned — roll back if corruptedState bucket config: versioning: on ← backup every state change encryption: CMEK ← encrypted with our KMS key uniform access: on ← no ACLs, just IAM public access: blocked ← never public
Common Interview Questions on This Topic
Q: Why Terraform over gcloud CLI scripts?
“Scripts are imperative — they tell you HOW to create something. Terraform is declarative — you tell it WHAT you want. If a script fails halfway, you have partial infrastructure and unclear state. Terraform tracks state, knows what exists, and only changes what’s different. Scripts don’t handle drift — Terraform can detect and fix it. Also Terraform plans show you exactly what will change before it happens — a gcloud script gives you no preview.”
Q: How do you handle Terraform state locking?
“GCS backend supports state locking natively via Cloud Storage object locking. When Terraform runs, it writes a lock file. If two applies run simultaneously, the second one detects the lock and fails with a clear error rather than corrupting state. We also set a timeout so locks don’t get stuck if a pipeline dies mid-run.”
Q: What happens if a node pool update requires node replacement?
“GKE handles this through surge upgrades — configured as max_surge=1, max_unavailable=0. It adds one new node with the new configuration, waits for it to be ready, drains one old node, then repeats. With PodDisruptionBudgets set on workloads, no service goes below its minimum replicas during the process. Zero downtime.”
upgrade_settings { strategy = "SURGE" max_surge = 1 # add 1 extra node during upgrade max_unavailable = 0 # never reduce below desired count}
Q: How do you manage secrets in your GKE workloads?
“Three layers. First, GCP Secret Manager stores the actual secrets — never in code or Terraform variables. Second, External Secrets Operator syncs them from Secret Manager into Kubernetes secrets automatically, with a 1-hour refresh. Third, pods reference Kubernetes secrets as environment variables or volume mounts — never as plain text in YAML. The whole chain is encrypted — KMS at rest in etcd, TLS in transit.”
Q: How do you handle cluster upgrades?
“We’re on the REGULAR release channel — Google automatically upgrades the control plane. For node pools, auto_upgrade=true handles it during our maintenance window, Saturday to Sunday 2-6 AM. We have a maintenance exclusion for high-traffic periods like Black Friday. The surge upgrade strategy ensures zero downtime. We test new versions in dev first since dev is on RAPID channel — so dev gets updates weeks before production.”
Q: What would you do differently if starting again?
“Two things. First, I’d use Terraform workspaces or Terragrunt from day one to reduce the environment config duplication — our three tfvars files have a lot of overlap. Second, I’d implement drift detection earlier — a daily GitHub Actions cron job running terraform plan and alerting if it detects changes. We added it later but it should have been day one because manual changes to the cluster are the biggest source of incidents.”
One-Line Summaries for Each Component
If interviewer asks "explain X in one sentence":VPC: "Private network that isolates our cluster from the public internet"Private cluster: "Nodes have no public IPs — attackers can't reach them directly even if they find them"Workload Identity:"Pods prove their identity cryptographically — no key files that can be stolen"KMS encryption: "Even if someone steals etcd, they can't read our secrets without the encryption key"Node pools: "Different machine types for different jobs — spot VMs for batch saves 40% on cost"Cluster Autoscaler:"Automatically adds nodes when pods can't schedule — removes them when idle"OIDC auth: "GitHub proves who it is to GCP without storing any credentials"Remote state: "Single source of truth for what Terraform thinks exists in the cloud"Modules: "Reusable, tested building blocks so every team gets secure infra without knowing the details"Release channel: "Google auto-upgrades our cluster on a schedule we control"
The Killer Answer Structure
When asked “Tell me about your GKE Terraform setup”, use this structure:
30-second version:"I built a modular Terraform platform that provisions production-grade GKE clusters — private networking, Workload Identity, KMS encryption — all by default. Provisioning time went from 4 hours to 12 minutes, and config drift dropped from hundreds of incidents per month to near zero."2-minute version:Add: module structure, three node pools, CI/CD pipeline, OIDC auth, one specific technical challenge you solved5-minute version:Add: specific decisions you made and why, alternatives you considered, what you'd do differently, metrics
The key is always anchor to outcomes — faster provisioning, better security posture, reduced drift, lower cost. Interviewers remember stories and numbers, not YAML.