Confluent Kafka

Deploys a Confluent Cloud Kafka cluster with configurable cluster type, multi-zone availability, and optional private networking. Supports Basic, Standard, Enterprise, and Dedicated cluster tiers across AWS, Azure, and GCP regions.

What Gets Created

When you deploy a ConfluentKafka resource, OpenMCF provisions:

• Confluent Cloud Kafka Cluster — a confluent_kafka_cluster resource of the specified type (Basic, Standard, Enterprise, or Dedicated), placed in the given cloud provider region and associated with a Confluent Cloud environment
• Network Association — created only when networkConfig is provided, associates the cluster with a pre-existing Confluent Cloud network for private connectivity (PrivateLink on AWS, Private Link on Azure, Private Service Connect on GCP)

Prerequisites

• Confluent Cloud credentials configured via environment variables (CONFLUENT_CLOUD_API_KEY, CONFLUENT_CLOUD_API_SECRET) or OpenMCF provider config
• A Confluent Cloud environment — the environmentId of an existing environment where the cluster will be created
• A Confluent Cloud network if enabling private networking via networkConfig — must be pre-created in the same environment
• Sufficient CKU quota if deploying a Dedicated cluster

Quick Start

Create a file confluent-kafka.yaml:

apiVersion: confluent.openmcf.org/v1
kind: ConfluentKafka
metadata:
  name: my-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.ConfluentKafka.my-kafka
spec:
  cloud: AWS
  region: us-east-2
  availability: SINGLE_ZONE
  environmentId: env-abc123

Deploy:

openmcf apply -f confluent-kafka.yaml

This creates a Standard Kafka cluster in a single availability zone on AWS us-east-2, associated with the specified Confluent Cloud environment.

Configuration Reference

Required Fields

Optional Fields

Examples

Basic Development Cluster

A low-cost single-zone cluster for development and testing:

apiVersion: confluent.openmcf.org/v1
kind: ConfluentKafka
metadata:
  name: dev-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.ConfluentKafka.dev-kafka
spec:
  cloud: GCP
  region: us-central1
  availability: SINGLE_ZONE
  environmentId: env-dev456
  clusterType: BASIC
  displayName: "Dev Kafka Cluster"

Standard Multi-Zone Production Cluster

A production-grade cluster with multi-zone availability for high availability:

apiVersion: confluent.openmcf.org/v1
kind: ConfluentKafka
metadata:
  name: prod-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.ConfluentKafka.prod-kafka
spec:
  cloud: AWS
  region: us-east-1
  availability: MULTI_ZONE
  environmentId: env-prod789
  clusterType: STANDARD
  displayName: "Production Event Bus"

Dedicated Cluster with Private Networking

A single-tenant Dedicated cluster with provisioned capacity and private network connectivity, suitable for regulated workloads requiring network isolation:

apiVersion: confluent.openmcf.org/v1
kind: ConfluentKafka
metadata:
  name: secure-kafka
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.ConfluentKafka.secure-kafka
spec:
  cloud: AZURE
  region: eastus
  availability: MULTI_ZONE
  environmentId: env-secure012
  clusterType: DEDICATED
  dedicatedConfig:
    cku: 2
  networkConfig:
    networkId: n-abc123
  displayName: "Secure Event Platform"

Stack Outputs

After deployment, the following outputs are available in status.outputs:

Related Components

No other OpenMCF components have direct foreign key references to ConfluentKafka. This component is typically deployed alongside application workloads that produce or consume Kafka messages.