Organizing GitOps: The App of Apps Structure

techhadoop kubernetes, OCP

To manage multiple global clusters while enforcing strict governance and standards, you need a highly organized GitOps directory structure. The industry standard pattern for this is the “App of Apps” pattern or ArgoCD ApplicationSets, combined with Kustomize to handle environmental differences without duplicating code.

Here is the blueprint for structuring a production-ready GitOps repository designed to scale to dozens of clusters across the globe.

1. The Global GitOps Directory Layout

Plaintext

├── clusters/                      # The Entry Point for each physical cluster
│   ├── production-us-east/
│   │   ├── core-platform/         # Core operators, security, networking
│   │   └── applications/          # App workloads for this cluster
│   └── development-eu-west/
│       ├── core-platform/
│       └── applications/
│
├── infrastructure/                # Base manifests shared across the enterprise
│   ├── ingress-controllers/
│   ├── service-mesh/
│   └── storage-classes/
│
├── governance/                    # Policy-as-Code and Security configurations
│   ├── Gatekeeper-policies/
│   ├── network-policies/
│   └── rbac/
│       ├── base/
│       └── overlays/
│           ├── production/        # Strict permissions
│           └── development/       # More relaxed permissions
│
└── tenants/                       # Developer team declarations (Quotas, Namespaces)
    ├── team-alpha/
    └── team-beta/

2. Deep Dive: How the Layers Work

The clusters/ Directory (The “What Goes Where” Layer)

This is the only directory that knows about specific physical infrastructure. If you spin up a new cluster in Tokyo, you simply add a new folder here: clusters/production-ap-northeast/.

  • Inside this folder, you have an ArgoCD root file that points back to the shared infrastructure/, governance/, and tenants/ directories.
The governance/ Directory (The “Guardrails” Layer)

This is your centralized security vault. Because it is separate from the application code, your Security and Compliance teams can own this directory.

  • By using Kustomize overlays, you can enforce strict, zero-trust network policies in the production/ overlay, while allowing a looser network policy configuration in the development/ overlay so developers can debug easily.
The tenants/ Directory (The “Onboarding” Layer)

When a new development team joins the organization, they don’t get cluster-admin rights to create namespaces. Instead, they submit a Pull Request to this directory.

  • Their YAML file defines their Namespace, their ResourceQuota limits, and their team’s access group (e.g., Okta group mapping). Once the PR is approved and merged, ArgoCD automatically provisions their environment across all global clusters.

3. Managing Differences using Kustomize

The biggest trap in multi-cluster management is copying and pasting YAML files for different environments. If you copy a manifest for Dev and paste it for Prod, they will eventually drift out of sync.

Instead, use Kustomize to keep a single “Base” manifest and inject environment-specific “Overlays.”

Example: The Base Resource Quota (tenants/team-alpha/base/quota.yaml)

YAML

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
spec:
  hard:
    pods: "10" # Default safe limit
Example: The Production Patch (tenants/team-alpha/overlays/production/patch.yaml)

In production, Team Alpha needs a much bigger footprint. Kustomize handles this by overriding just the specific value:

YAML

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
spec:
  hard:
    pods: "100" # Production scale upgrade

4. The Global Reconciliation Workflow

  1. The Change: A Senior Platform Engineer wants to roll out a new security policy globally. They create a branch, update the governance/Gatekeeper-policies/ directory, and open a Pull Request.
  2. The Validation: Automated CI pipelines (GitHub Actions/GitLab CI) run kube-linter and test the YAML syntax to ensure there are no configuration errors.
  3. The Approval: The Security Team reviews and merges the Pull Request into the main branch.
  4. The Deployment: ArgoCD or Red Hat ACM detects the change in the main branch. Within seconds, it pushes the new policy out to every single cluster registered in the global fleet, whether it’s in AWS, Azure, or on-premises.

Summary Best Practices for GitOps

  • Trunk-Based Development: Use a single main branch as the source of truth for your infrastructure. Avoid creating separate branches for dev, stage, and prod networks, as this leads to merge hell and environment drift. Use directory structures (overlays) instead.
  • Automated Pruning: Enable prune: true in your GitOps engine. If someone manually deletes a security policy using the CLI, the GitOps controller will immediately catch it and recreate it from the Git template.
  • No Secrets in Git: Never store passwords, TLS certificates, or database credentials in this repository. Use a GitOps-compatible secret provider like HashiCorp Vault, AWS Secrets Manager, or Sealed Secrets to inject sensitive data dynamically.

To scale this setup across dozens of clusters without manually writing a configuration file for every single one, you use an ArgoCD ApplicationSet.

An ApplicationSet uses a “Generator” to scan your Git repository (or your cluster API) and dynamically generate standard ArgoCD Applications on the fly. If you add a new cluster folder to your repository, the ApplicationSet notices it and automatically provisions that cluster without any human intervention.

Here is the production-ready manifest that connects our global directory structure together.

The Global Infrastructure ApplicationSet

This manifest uses the Git Generator. It tells ArgoCD to look inside the clusters/ directory, find every subfolder, and build a deployment pipeline for it.

YAML

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: global-infrastructure-fleet
  namespace: openshift-gitops
spec:
  generators:
    - git:
        repoURL: 'https://github.com/your-enterprise/openshift-gitops-infra.git'
        revision: HEAD
        # Directories matching this pattern will trigger a cluster deployment
        directories:
          - path: 'clusters/*'
  template:
    metadata:
      # Automatically names the application based on the folder name (e.g., "production-us-east-infra")
      name: '{{path.basename}}-infra'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/your-enterprise/openshift-gitops-infra.git'
        targetRevision: HEAD
        # Points directly to the 'core-platform' folder inside each cluster's directory
        path: '{{path}}/core-platform'
      destination:
        # Dynamically targets the correct cluster URL based on the folder configuration
        server: 'https://kubernetes.default.svc' # Or use an element from a cluster secret
        namespace: openshift-gitops
      syncPolicy:
        automated:
          prune: true      # Automatically delete resources if they are removed from Git
          selfHeal: true   # Overwrite manual changes if someone drifts from the Git standard
        syncOptions:
          - CreateNamespace=true # Create target namespaces if they don't exist yet

How This Works in Production

1. The Dynamic Substitution ({{path.basename}})

The magic lies in the {{path.basename}} template variable. If your repository contains the folders clusters/production-us-east and clusters/development-eu-west, the ApplicationSet generator expands that single block of code into two distinct, active ArgoCD Applications:

  • production-us-east-infra
  • development-eu-west-infra
2. The Core-Platform “App of Apps”

The ApplicationSet points to {{path}}/core-platform. Inside that folder, you place a kustomization.yaml file that links back to your global shared standards. It acts as the anchor that pulls in all your required cluster tools:

YAML

# Example contents of clusters/production-us-east/core-platform/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../infrastructure/ingress-controllers/base
  - ../../../infrastructure/storage-classes/base
  - ../../../governance/Gatekeeper-policies/overlays/production
  - ../../../governance/rbac/overlays/production

Security Guardrails for Global Fleet Management

When controlling a global fleet from a single ApplicationSet, a single broken YAML file could theoretically disrupt every cluster simultaneously. Implement these operational safety controls to prevent widespread issues:

  • Progressive Rollouts (Rolling Upgrades): You can add a strategy block to the ApplicationSet to update clusters in waves. For example, mandate that changes must successfully sync to development-eu-west and pass health checks before the engine is permitted to apply the updates to production-us-east.
  • Strict Pull Request Testing: Use your CI pipeline (GitHub Actions or GitLab CI) to run validation testing on every commit to the repository:Bash# Example CI validation commands kustomize build clusters/production-us-east/core-platform/ > /dev/null kube-linter lint clusters/ If Kustomize cannot compile the manifests properly or if a linting rule is broken, the pull request is blocked and cannot be merged into the main branch.

This completes the entire architectural pipeline: from raw physical networking and eBPF data routing, through namespace security and identity governance, up to a fully automated global multi-cluster GitOps engine.

Leave a Reply