OpenMCF logoOpenMCF

Loading...

Kubernetes Elasticsearch

Deploys an Elasticsearch cluster on Kubernetes using the Elastic Cloud on Kubernetes (ECK) operator, with optional Kibana, configurable data persistence via PersistentVolumeClaims, and optional external access through Gateway API ingress with automatic TLS certificate provisioning and HTTP-to-HTTPS redirect.

What Gets Created

When you deploy a KubernetesElasticsearch resource, OpenMCF provisions:

  • Namespace — created only when createNamespace is true
  • Elasticsearch Cluster — an ECK-managed Elasticsearch custom resource (v8.15.0) with configurable replicas, resource limits, and node roles (master, data, ingest)
  • Persistent Volumes — when persistenceEnabled is true, a PersistentVolumeClaim (ReadWriteOnce) is attached to each Elasticsearch pod for durable data storage
  • Kibana Instance — when kibana.enabled is true, an ECK-managed Kibana custom resource connected to the Elasticsearch cluster with configurable replicas and resource limits
  • Password Secret — an auto-generated Kubernetes Secret containing the elastic user password, created by the ECK operator
  • TLS Certificate — when any ingress is enabled, a cert-manager Certificate resource with a ClusterIssuer derived from the ingress hostname domain
  • Gateway + HTTPRoutes (Elasticsearch) — when Elasticsearch ingress is enabled, a Gateway API Gateway and HTTPRoute pair for HTTPS traffic plus an HTTP-to-HTTPS redirect route
  • Gateway + HTTPRoutes (Kibana) — when Kibana ingress is enabled, a separate Gateway and HTTPRoute pair for Kibana with the same HTTPS and redirect setup

Prerequisites

  • Kubernetes credentials configured via environment variables or OpenMCF provider config
  • A Kubernetes namespace that already exists, or set createNamespace to true
  • ECK operator installed in the cluster (manages Elasticsearch and Kibana custom resources)
  • A StorageClass available in the cluster if enabling persistence (most managed Kubernetes clusters provide a default)
  • Istio ingress gateway running in the istio-ingress namespace if enabling ingress (used as the Gateway API implementation)
  • cert-manager installed with a ClusterIssuer matching the ingress hostname domain if enabling ingress
  • external-dns running in the cluster if enabling ingress with a hostname

Quick Start

Create a file elasticsearch.yaml:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesElasticsearch
metadata:
  name: my-es
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.KubernetesElasticsearch.my-es
spec:
  namespace: search
  createNamespace: true

Deploy:

openmcf apply -f elasticsearch.yaml

This creates a single-node Elasticsearch 8.15.0 cluster with persistence enabled, a 1Gi PersistentVolumeClaim, default resource limits (1000m CPU, 1Gi memory), Kibana enabled with a single replica, and an auto-generated password stored in a Kubernetes Secret.

Configuration Reference

Required Fields

FieldTypeDescriptionValidation
namespacestringKubernetes namespace for the Elasticsearch deployment. Can reference a KubernetesNamespace resource via valueFrom.Required

Optional Fields

FieldTypeDefaultDescription
targetCluster.clusterKindenum—Kubernetes cluster kind. Valid values: AwsEksCluster, GcpGkeCluster, AzureAksCluster, DigitalOceanKubernetesCluster, CivoKubernetesCluster.
targetCluster.clusterNamestring—Name of the target Kubernetes cluster in the same environment.
createNamespaceboolfalseWhen true, creates the namespace before deploying resources.
elasticsearch.container.replicasint321Number of Elasticsearch pods to deploy. Each pod runs master, data, and ingest roles.
elasticsearch.container.resources.limits.cpustring1000mMaximum CPU allocation for each Elasticsearch pod.
elasticsearch.container.resources.limits.memorystring1GiMaximum memory allocation for each Elasticsearch pod.
elasticsearch.container.resources.requests.cpustring50mMinimum guaranteed CPU for each Elasticsearch pod.
elasticsearch.container.resources.requests.memorystring100MiMinimum guaranteed memory for each Elasticsearch pod.
elasticsearch.container.persistenceEnabledbooltrueEnables persistent storage for Elasticsearch data. When enabled, data is persisted to a PersistentVolumeClaim and restored on pod restart.
elasticsearch.container.diskSizestring1GiSize of the PersistentVolumeClaim attached to each Elasticsearch pod. Required when persistenceEnabled is true. Must be a valid Kubernetes quantity (e.g., 1Gi, 10Gi). Cannot be modified after creation.
elasticsearch.ingress.enabledboolfalseCreates a Gateway API Gateway and HTTPRoutes exposing Elasticsearch on port 9200 with TLS termination.
elasticsearch.ingress.hostnamestring—Hostname for external access (e.g., elasticsearch.example.com). Configured automatically via external-dns. Required when elasticsearch.ingress.enabled is true.
kibana.enabledbooltrueDeploys a Kibana instance connected to the Elasticsearch cluster.
kibana.container.replicasint321Number of Kibana pods to deploy.
kibana.container.resources.limits.cpustring1000mMaximum CPU allocation for each Kibana pod.
kibana.container.resources.limits.memorystring1GiMaximum memory allocation for each Kibana pod.
kibana.container.resources.requests.cpustring50mMinimum guaranteed CPU for each Kibana pod.
kibana.container.resources.requests.memorystring100MiMinimum guaranteed memory for each Kibana pod.
kibana.ingress.enabledboolfalseCreates a Gateway API Gateway and HTTPRoutes exposing Kibana on port 5601 with TLS termination.
kibana.ingress.hostnamestring—Hostname for external Kibana access (e.g., kibana.example.com). Configured automatically via external-dns. Required when kibana.ingress.enabled is true.

Examples

Development Elasticsearch without Persistence

A lightweight Elasticsearch instance for development with persistence disabled, Kibana enabled, and reduced resources:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesElasticsearch
metadata:
  name: dev-es
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.KubernetesElasticsearch.dev-es
spec:
  namespace: dev
  createNamespace: true
  elasticsearch:
    container:
      replicas: 1
      persistenceEnabled: false
      resources:
        limits:
          cpu: "500m"
          memory: "512Mi"
        requests:
          cpu: "100m"
          memory: "256Mi"
  kibana:
    enabled: true
    container:
      replicas: 1
      resources:
        limits:
          cpu: "500m"
          memory: "512Mi"
        requests:
          cpu: "100m"
          memory: "256Mi"

Production Elasticsearch with Increased Storage

A production Elasticsearch cluster with multiple replicas, larger disk allocation, and higher resource limits:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesElasticsearch
metadata:
  name: prod-es
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.KubernetesElasticsearch.prod-es
spec:
  namespace: production
  elasticsearch:
    container:
      replicas: 3
      resources:
        limits:
          cpu: "4000m"
          memory: "8Gi"
        requests:
          cpu: "1000m"
          memory: "4Gi"
      persistenceEnabled: true
      diskSize: "100Gi"
  kibana:
    enabled: true
    container:
      replicas: 2
      resources:
        limits:
          cpu: "2000m"
          memory: "2Gi"
        requests:
          cpu: "500m"
          memory: "1Gi"

Elasticsearch with External Ingress

Elasticsearch and Kibana exposed outside the cluster via Gateway API with TLS termination and automatic DNS:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesElasticsearch
metadata:
  name: shared-es
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.KubernetesElasticsearch.shared-es
spec:
  namespace: shared-services
  elasticsearch:
    container:
      replicas: 3
      resources:
        limits:
          cpu: "4000m"
          memory: "8Gi"
        requests:
          cpu: "1000m"
          memory: "4Gi"
      persistenceEnabled: true
      diskSize: "200Gi"
    ingress:
      enabled: true
      hostname: elasticsearch.example.com
  kibana:
    enabled: true
    container:
      replicas: 2
      resources:
        limits:
          cpu: "2000m"
          memory: "2Gi"
        requests:
          cpu: "500m"
          memory: "1Gi"
    ingress:
      enabled: true
      hostname: kibana.example.com

Using Foreign Key References

Reference an OpenMCF-managed namespace instead of hardcoding the name:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesElasticsearch
metadata:
  name: search
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.KubernetesElasticsearch.search
spec:
  namespace:
    valueFrom:
      kind: KubernetesNamespace
      name: search-namespace
      field: spec.name
  elasticsearch:
    container:
      replicas: 3
      persistenceEnabled: true
      diskSize: "50Gi"
  kibana:
    enabled: true

Stack Outputs

After deployment, the following outputs are available in status.outputs:

OutputTypeDescription
namespacestringKubernetes namespace where Elasticsearch is deployed
elasticsearch.servicestringKubernetes Service name for Elasticsearch (format: {name}-es-http)
elasticsearch.portForwardCommandstringkubectl port-forward command for local access on port 9200
elasticsearch.kubeEndpointstringCluster-internal FQDN (e.g., my-es-es-http.search.svc.cluster.local)
elasticsearch.externalHostnamestringPublic hostname for external access, only set when Elasticsearch ingress is enabled
elasticsearch.usernamestringElasticsearch username (always elastic)
elasticsearch.passwordSecret.namestringName of the Kubernetes Secret containing the Elasticsearch password (format: {name}-es-elastic-user)
elasticsearch.passwordSecret.keystringKey within the password Secret (always elastic)
kibana.servicestringKubernetes Service name for Kibana (format: {name}-kb-http)
kibana.portForwardCommandstringkubectl port-forward command for local Kibana access on port 5601
kibana.kubeEndpointstringCluster-internal FQDN for Kibana (e.g., my-es-kb-http.search.svc.cluster.local)
kibana.externalHostnamestringPublic hostname for external Kibana access, only set when Kibana ingress is enabled

Related Components

  • KubernetesNamespace — provides the target namespace via valueFrom reference
  • KubernetesDeployment — application deployments that query Elasticsearch for search or analytics
  • KubernetesSecret — manage additional secrets consumed by Elasticsearch clients

Next article

Kubernetes External DNS

Kubernetes External DNS Deploys ExternalDNS on Kubernetes using the official Helm chart (external-dns v1.19.0) from kubernetes-sigs, with support for Google Cloud DNS (GKE), AWS Route53 (EKS), Azure DNS (AKS), and Cloudflare as DNS providers, automatic ServiceAccount creation with workload-identity annotations, optional namespace creation, and configurable ExternalDNS and Helm chart versions. What Gets Created When you deploy a KubernetesExternalDns resource, OpenMCF provisions: Namespace —...
Read next article
Presets
3 ready-to-deploy configurationsView presets →