OpenMCF logoOpenMCF

Loading...

Kustomize Integration

Kustomize lets you manage variations of OpenMCF manifests without duplication. Instead of maintaining separate manifest files for dev, staging, and production, you maintain one base manifest and environment-specific overlays that patch the base.

For the conceptual overview of manifest sources (including Kustomize), see Manifests. For flag details, see CLI Reference.

manifests/database/
|-- base/
|   \-- database.yaml      # Shared configuration
\-- overlays/
    |-- dev/
    |-- staging/
    \-- prod/               # Environment-specific patches

OpenMCF integrates Kustomize as a Go library (sigs.k8s.io/kustomize), not as an external binary. The --kustomize-dir and --overlay flags trigger Kustomize to build the final manifest at deployment time.

Quick Start

1. Create Base Manifest

mkdir -p services/api/kustomize/base

services/api/kustomize/base/deployment.yaml:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesDeployment
metadata:
  name: api
spec:
  container:
    image:
      repo: myapp/api
      tag: latest
    replicas: 1
    resources:
      limits:
        cpu: 500m
        memory: 512Mi

services/api/kustomize/base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

2. Create Environment Overlay

services/api/kustomize/overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

patches:
  - path: patch.yaml

services/api/kustomize/overlays/prod/patch.yaml:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesDeployment
metadata:
  name: api
spec:
  container:
    image:
      tag: v1.0.0
    replicas: 3
    resources:
      limits:
        cpu: 2000m
        memory: 4Gi

3. Deploy with OpenMCF

# Deploy to production
openmcf pulumi up \
  --kustomize-dir services/api/kustomize \
  --overlay prod

# Deploy to dev
openmcf pulumi up \
  --kustomize-dir services/api/kustomize \
  --overlay dev

What happens:

  1. OpenMCF runs kustomize build services/api/kustomize/overlays/prod
  2. Merges base + prod overlay into final manifest
  3. Validates the result
  4. Deploys using Pulumi or OpenTofu

Directory Structure

Standard Layout

<service-name>/kustomize/
|-- base/
|   |-- kustomization.yaml          # Base kustomization config
|   \-- <resource>.yaml             # Base resource definition
\-- overlays/
    |-- dev/
    |   |-- kustomization.yaml      # Dev environment config
    |   \-- patch.yaml              # Dev-specific patches
    |-- staging/
    |   |-- kustomization.yaml
    |   \-- patch.yaml
    \-- prod/
        |-- kustomization.yaml
        \-- patch.yaml

Example: Complete Service

backend/services/api/kustomize/
|-- base/
|   |-- kustomization.yaml
|   |-- deployment.yaml
|   \-- database.yaml
\-- overlays/
    |-- dev/
    |   |-- kustomization.yaml
    |   |-- deployment-patch.yaml
    |   \-- database-patch.yaml
    |-- staging/
    |   |-- kustomization.yaml
    |   |-- deployment-patch.yaml
    |   \-- database-patch.yaml
    \-- prod/
        |-- kustomization.yaml
        |-- deployment-patch.yaml
        \-- database-patch.yaml

Creating Patches

Strategic Merge Patches

The most common approach - specify only the fields you want to change:

Base:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesPostgres
metadata:
  name: app-database
spec:
  container:
    replicas: 1
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
    diskSize: 10Gi

Prod Patch (overlays/prod/patch.yaml):

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesPostgres
metadata:
  name: app-database
spec:
  container:
    replicas: 3              # Override
    resources:
      limits:
        cpu: 2000m           # Override
        memory: 4Gi          # Override
    diskSize: 100Gi          # Override

Result: Base + patch merged = 3 replicas, 2000m CPU, 4Gi memory, 100Gi disk.

JSON 6902 Patches

For more complex changes:

overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

patches:
  - target:
      kind: KubernetesPostgres
      name: app-database
    patch: |-
      - op: replace
        path: /spec/container/replicas
        value: 3
      - op: add
        path: /metadata/labels/environment
        value: production

Common Patterns

Pattern 1: Environment-Specific Resources

Different instance sizes per environment:

Dev (small, cheap):

spec:
  container:
    replicas: 1
    resources:
      limits:
        cpu: 500m
        memory: 512Mi

Prod (large, resilient):

spec:
  container:
    replicas: 5
    resources:
      limits:
        cpu: 2000m
        memory: 4Gi

Pattern 2: Environment-Specific Labels

Add labels for cost tracking:

overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

commonLabels:
  environment: production
  cost-center: engineering
  team: backend

Pattern 3: Environment-Specific Images

overlays/dev/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

images:
  - name: myapp/api
    newTag: latest          # Dev uses latest

patches:
  - path: dev-patch.yaml

overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

images:
  - name: myapp/api
    newTag: v1.2.3          # Prod uses specific version

patches:
  - path: prod-patch.yaml

Pattern 4: Shared Configuration + Environment Patches

base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
  app: api
  managed-by: openmcf

resources:
  - deployment.yaml
  - database.yaml
  - cache.yaml

Each resource in base defines shared configuration, overlays patch for environment needs.

Workflow Examples

Deploying to Multiple Environments

# Deploy to dev
openmcf pulumi up \
  --kustomize-dir services/api/kustomize \
  --overlay dev \
  --yes

# Test in dev...

# Deploy to staging
openmcf pulumi up \
  --kustomize-dir services/api/kustomize \
  --overlay staging \
  --yes

# Test in staging...

# Deploy to production (with review)
openmcf pulumi preview \
  --kustomize-dir services/api/kustomize \
  --overlay prod

# Review changes...

openmcf pulumi up \
  --kustomize-dir services/api/kustomize \
  --overlay prod

Combining Kustomize with --set Overrides

# Kustomize overlay + runtime override
openmcf pulumi up \
  --kustomize-dir services/api/kustomize \
  --overlay prod \
  --set spec.container.image.tag=v1.2.4

Order of precedence:

  1. Base manifest
  2. Overlay patches applied
  3. --set overrides applied last (highest priority)

Preview Built Manifest

# See what Kustomize generates (useful for debugging)
cd services/api/kustomize
kustomize build overlays/prod

# Or let OpenMCF build and show it
openmcf load-manifest \
  --kustomize-dir services/api/kustomize \
  --overlay prod

For using Kustomize in CI/CD pipelines with branch-based overlay selection, see CI/CD Integration.

Advanced Techniques

Multiple Bases

Useful for shared components:

common/
\-- base/
    |-- kustomization.yaml
    \-- shared-config.yaml

service-a/kustomize/
\-- overlays/
    \-- prod/
        |-- kustomization.yaml  # References ../../../common/base
        \-- patch.yaml

service-b/kustomize/
\-- overlays/
    \-- prod/
        |-- kustomization.yaml  # Also references ../../../common/base
        \-- patch.yaml

Components (Reusable Pieces)

components/monitoring/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
  - path: add-monitoring.yaml

Use in overlay:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

components:
  - ../../../components/monitoring

Generating ConfigMaps from Files

overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

configMapGenerator:
  - name: app-config
    files:
      - config/prod.yaml
      - config/secrets.encrypted

Troubleshooting

Error: "no such file or directory"

Problem: Kustomize can't find referenced files.

Solution:

# Check file paths in kustomization.yaml
# Ensure paths are relative to kustomization.yaml location

# Verify structure
ls -R services/api/kustomize/

Error: "kustomization.yaml not found"

Problem: Missing kustomization.yaml in overlay.

Solution:

# Create kustomization.yaml
cat > services/api/kustomize/overlays/prod/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base
EOF

List Fields Replaced Instead of Merged

Problem: Your overlay adds entries to a list field (e.g., env.variables, ports), but the overlay's list replaces the base list entirely instead of merging.

Cause: Kustomize only knows how to merge lists by key for built-in Kubernetes types. For OpenMCF custom resource types (KubernetesDeployment, KubernetesCronJob, etc.), it has no schema and falls back to replacing lists wholesale.

Solution: Use openmcf kustomize init to generate and wire up the OpenAPI schema that teaches kustomize how to merge OpenMCF lists correctly. See OpenAPI Schema for List Merging below.

Patch Not Applied

Problem: Your patch isn't affecting the final output.

Solution:

# 1. Verify patch is listed in kustomization.yaml
cat overlays/prod/kustomization.yaml

# 2. Check patch targets correct resource
# - apiVersion must match
# - kind must match
# - metadata.name must match

# 3. Test kustomize build directly
cd services/api/kustomize
kustomize build overlays/prod

Wrong Overlay Applied

Problem: Deployed dev config to prod (or vice versa).

Solution:

# Always verify overlay before deploying
openmcf pulumi preview \
  --kustomize-dir services/api/kustomize \
  --overlay prod  # Double-check this!

# Use explicit confirmation in CI/CD
if [ "$OVERLAY" != "prod" ]; then
  echo "Deploying to $OVERLAY"
  openmcf pulumi up --kustomize-dir ... --overlay $OVERLAY --yes
else
  echo "Production deployment - manual approval required"
  openmcf pulumi up --kustomize-dir ... --overlay $OVERLAY
fi

OpenAPI Schema for List Merging

Kustomize uses strategic merge patch for overlays. For built-in Kubernetes types, it knows to merge list fields by key (e.g., containers by name, env by name). For OpenMCF custom resource types, kustomize has no schema and falls back to replacing lists entirely.

This means an overlay that adds one environment variable will replace the entire variables list from the base, losing all base entries.

OpenMCF solves this with a built-in schema generator that uses proto reflection to discover all list fields with merge keys across all 360+ cloud resource kinds.

Generating the Schema

# Print the universal schema to stdout
openmcf kustomize schema

# Write to a file
openmcf kustomize schema -o openmcf-schema.json

The schema covers every OpenMCF kind that has list fields which should merge by name — environment variables, secrets, ports, volume mounts, sidecars, and more. Kinds without merge-worthy fields are excluded automatically.

Initializing a Kustomize Directory

# Initialize a single service's _kustomize directory
openmcf kustomize init --dir ./services/api/_kustomize

# Scan a directory tree and initialize ALL _kustomize directories
openmcf kustomize init --scan ./services

The init command:

  1. Generates the universal schema
  2. Writes openmcf-schema.json at the _kustomize/ root
  3. Adds the openapi: reference to every overlay kustomization.yaml

After initialization, overlay kustomization files look like:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

openapi:
  path: ../../openmcf-schema.json

resources:
  - ../../base

patches:
  - path: patch.yaml

When to Re-run

Re-run openmcf kustomize init after upgrading OpenMCF to pick up schema changes from new or modified cloud resource kinds. The command is idempotent — it regenerates the schema file and skips overlays that already have the openapi: reference.

What Gets Merged

The schema declares x-kubernetes-patch-merge-key: "name" for list fields whose elements have a name field. Common examples:

KindField PathMerge Key
KubernetesDeploymentspec.container.app.env.variablesname
KubernetesDeploymentspec.container.app.env.secretsname
KubernetesDeploymentspec.container.app.portsname
KubernetesDeploymentspec.container.app.volumeMountsname
KubernetesDeploymentspec.container.sidecarsname
KubernetesCronJobspec.env.variablesname
KubernetesCronJobspec.env.secretsname
KubernetesJobspec.env.variablesname
KubernetesServicespec.portsname

Scalar fields (like replicas, cpu) and map fields (like configMaps) are unaffected — kustomize already handles those correctly without a schema.

Best Practices

1. Keep Base Minimal

# ✅ Good: Base has common configuration
base/deployment.yaml:
  name: api
  container:
    image:
      repo: myapp/api
    # No environment-specific values

# ❌ Bad: Base has production values
base/deployment.yaml:
  name: api
  container:
    replicas: 10  # Production-specific, doesn't belong in base

2. One Overlay Per Environment

# Good
overlays/
|-- dev/
|-- staging/
\-- prod/

# Confusing
overlays/
|-- dev-us-west/
|-- dev-eu-central/
|-- staging-us-west/
\-- ... (too many combinations)

3. Use Descriptive Patch Names

# Good
overlays/prod/
|-- kustomization.yaml
|-- resources-patch.yaml          # Increases resources
|-- replicas-patch.yaml            # Scales replicas
\-- monitoring-patch.yaml          # Adds monitoring

# Bad
overlays/prod/
|-- kustomization.yaml
|-- patch1.yaml
|-- patch2.yaml
\-- patch3.yaml

4. Version Control Everything

# ✅ Good: All kustomize files in Git
git add services/api/kustomize/
git commit -m "feat: add production overlay for API"

# ❌ Bad: Generated files, temp files committed
git add services/api/kustomize/overlays/prod/output.yaml  # Generated

5. Test Overlays Before Deploying

# ✅ Good: Preview before applying
kustomize build overlays/prod | less  # Review output
openmcf pulumi preview --kustomize-dir ... --overlay prod

# ⚠️ Risky: Deploy without review
openmcf pulumi up --kustomize-dir ... --overlay prod --yes

Complete Example

Here's a complete real-world example:

Directory Structure

backend/services/api/kustomize/
|-- base/
|   |-- kustomization.yaml
|   |-- deployment.yaml
|   |-- database.yaml
|   \-- redis.yaml
\-- overlays/
    |-- dev/
    |   |-- kustomization.yaml
    |   |-- deployment-patch.yaml
    |   |-- database-patch.yaml
    |   \-- redis-patch.yaml
    \-- prod/
        |-- kustomization.yaml
        |-- deployment-patch.yaml
        |-- database-patch.yaml
        \-- redis-patch.yaml

Files

base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
  app: api
  managed-by: openmcf

resources:
  - deployment.yaml
  - database.yaml
  - redis.yaml

base/deployment.yaml:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesDeployment
metadata:
  name: api
spec:
  container:
    image:
      repo: mycompany/api
      tag: latest
    replicas: 1
    resources:
      limits:
        cpu: 500m
        memory: 512Mi

overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

commonLabels:
  environment: production

images:
  - name: mycompany/api
    newTag: v1.0.0

patches:
  - path: deployment-patch.yaml
  - path: database-patch.yaml
  - path: redis-patch.yaml

overlays/prod/deployment-patch.yaml:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesDeployment
metadata:
  name: api
spec:
  container:
    replicas: 5
    resources:
      limits:
        cpu: 2000m
        memory: 4Gi

Deployment

# Deploy to production
openmcf pulumi up \
  --kustomize-dir backend/services/api/kustomize \
  --overlay prod

What's Next

  • Writing Manifests — Manifest structure and best practices
  • CI/CD Integration — Kustomize with GitHub Actions and GitLab CI
  • Advanced Usage — Combining Kustomize with --set overrides
  • State Backends — Per-environment state configuration
  • CLI Reference — Full openmcf kustomize schema and openmcf kustomize init usage
  • Official Kustomize Docs — Kustomize reference documentation

Next article

Advanced Usage

Advanced Usage Advanced techniques for power users and complex deployment scenarios. This page assumes familiarity with Writing Manifests, Kustomize Integration, and the CLI Reference. Runtime Value Overrides with --set The --set flag lets you override manifest values without editing files. Think of it like command-line arguments for your infrastructure. Basic Syntax Nested Fields Use dot notation to access nested fields: Multiple Overrides Repeat the --set flag multiple times: When to Use...
Read next article