OpenMCF logoOpenMCF

Loading...

Kubernetes Kafka

Deploys an Apache Kafka cluster on Kubernetes using the Strimzi operator, with Zookeeper-based coordination, SCRAM-SHA-512 authentication, optional Schema Registry (Confluent), optional Kafka UI (Kowl), per-broker persistent storage, and external access via TLS-terminated load balancers with automatic DNS management.

What Gets Created

When you deploy a KubernetesKafka resource, OpenMCF provisions:

  • Namespace — created only when createNamespace is true
  • Strimzi Kafka Cluster — a Kafka custom resource with configurable broker replicas, resource limits, JBOD persistent storage, simple authorization, and SCRAM-SHA-512 authentication on all listeners
  • Zookeeper Ensemble — co-deployed with the Kafka cluster, using persistent-claim storage for durable coordination state
  • Entity Operator — Strimzi Topic Operator and User Operator for declarative topic and user management
  • Admin User — a KafkaUser custom resource with SCRAM-SHA-512 credentials and super-user privileges, with the password stored in a Kubernetes Secret
  • Kafka Topics — one KafkaTopic custom resource per entry in kafkaTopics, each with configurable partitions, replicas, and topic-level settings
  • Schema Registry — a Confluent Schema Registry Deployment and ClusterIP Service, created only when schemaRegistryContainer.isEnabled is true; includes Gateway API ingress resources when ingress is enabled
  • Kafka UI (Kowl) — a Kowl Deployment, ConfigMap, and ClusterIP Service, created only when isDeployKafkaUi is true; includes Gateway API ingress resources when ingress is enabled
  • TLS Certificates — cert-manager Certificate resources for bootstrap server, schema registry, and Kowl hostnames, issued by a ClusterIssuer matching the ingress domain
  • Load Balancer Listeners — public and private Strimzi load balancer listeners with per-broker advertised hostnames and external-dns annotations, created only when ingress is enabled
  • Gateway API Resources — Gateway and HTTPRoute resources for Schema Registry and Kowl external access via Istio ingress

Prerequisites

  • Kubernetes credentials configured via environment variables or OpenMCF provider config
  • A Kubernetes namespace that already exists, or set createNamespace to true
  • Strimzi Kafka Operator installed in the cluster (provides Kafka, KafkaTopic, and KafkaUser CRDs)
  • cert-manager installed with a ClusterIssuer matching the ingress domain, required when ingress is enabled
  • external-dns running in the cluster if enabling ingress with hostnames
  • Istio ingress gateway deployed in the istio-ingress namespace, required for Schema Registry and Kowl external access
  • A StorageClass available in the cluster for broker and Zookeeper persistent volumes

Quick Start

Create a file kafka.yaml:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesKafka
metadata:
  name: my-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.KubernetesKafka.my-kafka
spec:
  namespace: kafka
  createNamespace: true

Deploy:

openmcf apply -f kafka.yaml

This creates a single-broker Kafka cluster with a single Zookeeper node, 1Gi persistent storage for each, SCRAM-SHA-512 authentication, an admin super-user, and the Kowl UI enabled by default.

Configuration Reference

Required Fields

FieldTypeDescriptionValidation
namespacestringKubernetes namespace for the Kafka deployment. Can reference a KubernetesNamespace resource via valueFrom.Required

Optional Fields

FieldTypeDefaultDescription
targetCluster.clusterKindenum—Kubernetes cluster kind. Valid values: AwsEksCluster, GcpGkeCluster, AzureAksCluster, DigitalOceanKubernetesCluster, CivoKubernetesCluster.
targetCluster.clusterNamestring—Name of the target Kubernetes cluster in the same environment.
createNamespaceboolfalseWhen true, creates the namespace before deploying resources.
kafkaTopicslist[]List of Kafka topics to create. Each entry accepts name, partitions, replicas, and config.
kafkaTopics[].namestring—Topic name. Must be 1-249 characters, start and end with an alphanumeric character, and contain only alphanumerics, ., _, or -.
kafkaTopics[].partitionsint321Number of partitions for the topic.
kafkaTopics[].replicasint321Number of replicas for the topic.
kafkaTopics[].configmap<string,string>See belowTopic-level configuration overrides. Defaults include cleanup.policy: delete, retention.ms: 604800000, max.message.bytes: 2097164, and others.
brokerContainer.replicasint321Number of Kafka broker pods.
brokerContainer.resources.limits.cpustring1000mMaximum CPU allocation for each broker pod.
brokerContainer.resources.limits.memorystring1GiMaximum memory allocation for each broker pod.
brokerContainer.resources.requests.cpustring50mMinimum guaranteed CPU for each broker pod.
brokerContainer.resources.requests.memorystring100MiMinimum guaranteed memory for each broker pod.
brokerContainer.diskSizestring1GiPersistent volume size for each broker. Must be a valid Kubernetes quantity (e.g., 1Gi, 30Gi).
zookeeperContainer.replicasint321Number of Zookeeper pods. Use 3 or more for high availability (Raft consensus).
zookeeperContainer.resources.limits.cpustring1000mMaximum CPU allocation for each Zookeeper pod.
zookeeperContainer.resources.limits.memorystring1GiMaximum memory allocation for each Zookeeper pod.
zookeeperContainer.resources.requests.cpustring50mMinimum guaranteed CPU for each Zookeeper pod.
zookeeperContainer.resources.requests.memorystring100MiMinimum guaranteed memory for each Zookeeper pod.
zookeeperContainer.diskSizestring1GiPersistent volume size for each Zookeeper node. Must be a valid Kubernetes quantity.
schemaRegistryContainer.isEnabledboolfalseDeploys a Confluent Schema Registry alongside Kafka.
schemaRegistryContainer.replicasint321Number of Schema Registry pods. Only applies when isEnabled is true.
schemaRegistryContainer.resources.limits.cpustring1000mMaximum CPU allocation for each Schema Registry pod.
schemaRegistryContainer.resources.limits.memorystring1GiMaximum memory allocation for each Schema Registry pod.
schemaRegistryContainer.resources.requests.cpustring50mMinimum guaranteed CPU for each Schema Registry pod.
schemaRegistryContainer.resources.requests.memorystring100MiMinimum guaranteed memory for each Schema Registry pod.
ingress.enabledboolfalseEnables external access via TLS-terminated load balancers for Kafka brokers, and Gateway API ingress for Schema Registry and Kowl.
ingress.hostnamestring—Base hostname for external access (e.g., kafka.example.com). Required when ingress.enabled is true. Used to derive broker, Schema Registry, and Kowl hostnames.
isDeployKafkaUibooltrueDeploys the Kowl web UI for browsing topics, consumer groups, and messages.

Examples

Development Kafka Cluster

A minimal single-broker Kafka cluster for development with reduced resources and no external access:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesKafka
metadata:
  name: dev-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.KubernetesKafka.dev-kafka
spec:
  namespace: dev-kafka
  createNamespace: true
  kafkaTopics:
    - name: events
    - name: notifications
  brokerContainer:
    replicas: 1
    resources:
      limits:
        cpu: "500m"
        memory: "512Mi"
      requests:
        cpu: "100m"
        memory: "256Mi"
    diskSize: "5Gi"
  zookeeperContainer:
    replicas: 1
    resources:
      limits:
        cpu: "250m"
        memory: "512Mi"
      requests:
        cpu: "50m"
        memory: "128Mi"
    diskSize: "1Gi"

Production Kafka with Schema Registry and Ingress

A multi-broker production cluster with Schema Registry, Kafka UI, TLS ingress, and increased storage:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesKafka
metadata:
  name: prod-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.KubernetesKafka.prod-kafka
spec:
  namespace: kafka-prod
  kafkaTopics:
    - name: orders
      partitions: 12
      replicas: 3
    - name: user-events
      partitions: 6
      replicas: 3
      config:
        cleanup.policy: compact
        retention.ms: "-1"
    - name: dead-letter
      partitions: 3
      replicas: 3
      config:
        retention.ms: "2592000000"
  brokerContainer:
    replicas: 3
    resources:
      limits:
        cpu: "4000m"
        memory: "8Gi"
      requests:
        cpu: "1000m"
        memory: "4Gi"
    diskSize: "100Gi"
  zookeeperContainer:
    replicas: 3
    resources:
      limits:
        cpu: "2000m"
        memory: "4Gi"
      requests:
        cpu: "500m"
        memory: "1Gi"
    diskSize: "10Gi"
  schemaRegistryContainer:
    isEnabled: true
    replicas: 2
    resources:
      limits:
        cpu: "2000m"
        memory: "2Gi"
      requests:
        cpu: "250m"
        memory: "512Mi"
  ingress:
    enabled: true
    hostname: kafka.prod.example.com
  isDeployKafkaUi: true

Using Foreign Key References

Reference an OpenMCF-managed namespace instead of hardcoding the name:

apiVersion: kubernetes.openmcf.org/v1
kind: KubernetesKafka
metadata:
  name: shared-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: staging.KubernetesKafka.shared-kafka
spec:
  namespace:
    valueFrom:
      kind: KubernetesNamespace
      name: messaging-ns
      field: spec.name
  kafkaTopics:
    - name: audit-log
      partitions: 6
      replicas: 2
      config:
        cleanup.policy: compact
        retention.ms: "-1"
  brokerContainer:
    replicas: 3
    resources:
      limits:
        cpu: "2000m"
        memory: "4Gi"
      requests:
        cpu: "500m"
        memory: "1Gi"
    diskSize: "50Gi"
  zookeeperContainer:
    replicas: 3
    diskSize: "5Gi"
  schemaRegistryContainer:
    isEnabled: true
  isDeployKafkaUi: true

Stack Outputs

After deployment, the following outputs are available in status.outputs:

OutputTypeDescription
namespacestringKubernetes namespace where Kafka is deployed
usernamestringSASL admin username for Kafka authentication
passwordSecret.namestringName of the Kubernetes Secret containing the admin password
passwordSecret.keystringKey within the password Secret (always password)
bootstrapServerExternalHostnamestringPublic hostname of the Kafka bootstrap server, only set when ingress is enabled
bootstrapServerInternalHostnamestringInternal (VPC-level) hostname of the Kafka bootstrap server, only set when ingress is enabled
schemaRegistryExternalUrlstringPublic HTTPS URL for the Schema Registry, empty when Schema Registry is not enabled
schemaRegistryInternalUrlstringInternal HTTPS URL for the Schema Registry, empty when Schema Registry is not enabled
kafkaUiExternalUrlstringPublic HTTPS URL for the Kowl Kafka UI, only set when isDeployKafkaUi is true and ingress is enabled

Related Components

  • KubernetesNamespace — provides the target namespace via valueFrom reference
  • KubernetesDeployment — application deployments that produce to or consume from Kafka topics
  • KubernetesPostgres — often deployed alongside Kafka for event-sourced architectures with a relational read store
  • KubernetesRedis — complementary caching layer for applications consuming Kafka events

Next article

Kubernetes Keycloak

Kubernetes Keycloak Deploys Keycloak on Kubernetes as an identity and access management solution. Provisions a Keycloak instance with configurable container resources, optional namespace creation, and optional external access through ingress with TLS. Keycloak provides single sign-on, identity brokering, user federation, and fine-grained authorization for applications and services. What Gets Created When you deploy a KubernetesKeycloak resource, OpenMCF provisions: Kubernetes Namespace —...
Read next article
Presets
3 ready-to-deploy configurationsView presets →