AWS Kinesis Firehose

Deploys an Amazon Kinesis Data Firehose delivery stream that captures, transforms, and delivers streaming data to S3, OpenSearch, HTTP endpoints, or Redshift. The component supports Direct PUT and Kinesis Data Stream sources, optional Lambda transformation, dynamic partitioning, and Parquet/ORC format conversion.

What Gets Created

When you deploy an AwsKinesisFirehose resource, OpenMCF provisions:

• Kinesis Firehose Delivery Stream — the core aws_kinesis_firehose_delivery_stream resource configured with the selected destination type
• Kinesis source configuration — created only when kinesisStreamSource is set, configures Firehose to consume from an existing Kinesis Data Stream with automatic checkpointing and retry
• Server-side encryption — created only when sseEnabled is true, encrypts data at rest in the delivery stream buffer using AWS-owned or customer-managed KMS keys (Direct PUT sources only)
• Extended S3 destination — primary S3 delivery with optional GZIP/Snappy compression, Lambda processing, dynamic partitioning, and Parquet/ORC format conversion via AWS Glue Data Catalog
• OpenSearch destination — direct indexing into an OpenSearch domain with configurable index rotation, VPC delivery, and S3 backup for failed documents
• HTTP endpoint destination — HTTPS delivery to any endpoint (Datadog, New Relic, Sumo Logic, custom APIs) with S3 backup for failed records
• Redshift destination — S3 staging followed by a Redshift COPY command for bulk data warehouse loading

Prerequisites

• AWS credentials configured via environment variables or OpenMCF provider config
• A destination resource — an S3 bucket, OpenSearch domain, HTTPS endpoint, or Redshift cluster depending on the chosen destination type
• An IAM role with permissions appropriate for the destination (S3 write, OpenSearch index, Redshift COPY, etc.)
• A Kinesis Data Stream if using Kinesis source mode instead of Direct PUT
• An AWS Glue Data Catalog database and table if enabling Parquet/ORC data format conversion
• VPC subnets and security groups if delivering to a VPC-deployed OpenSearch domain

Quick Start

Create a file firehose.yaml:

apiVersion: aws.openmcf.org/v1
kind: AwsKinesisFirehose
metadata:
  name: my-firehose
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.AwsKinesisFirehose.my-firehose
spec:
  region: us-east-1
  extendedS3:
    bucketArn: arn:aws:s3:::my-data-bucket
    roleArn: arn:aws:iam::123456789012:role/firehose-s3-role

Deploy:

openmcf apply -f firehose.yaml

This creates a Direct PUT delivery stream that writes raw records to S3 with no compression or transformation.

Configuration Reference

Required Fields

Exactly one destination must be configured. The destination type is ForceNew — changing it requires replacing the delivery stream.

Optional Fields

Extended S3 Destination Fields

OpenSearch Destination Fields

HTTP Endpoint Destination Fields

Redshift Destination Fields

Examples

Extended S3 Data Lake

GZIP-compressed delivery to S3 with timestamp-based prefixes and buffering tuned for throughput:

apiVersion: aws.openmcf.org/v1
kind: AwsKinesisFirehose
metadata:
  name: data-lake-firehose
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.AwsKinesisFirehose.data-lake-firehose
spec:
  region: us-east-1
  extendedS3:
    bucketArn: arn:aws:s3:::my-data-lake-bucket
    roleArn: arn:aws:iam::123456789012:role/firehose-s3-delivery-role
    prefix: events/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/
    compressionFormat: GZIP
    fileExtension: .json.gz
    buffering:
      intervalInSeconds: 120
      sizeInMbs: 64

OpenSearch Log Analytics

Indexes application logs into an OpenSearch domain with daily index rotation and S3 backup for failed documents. References the OpenSearch domain via valueFrom:

apiVersion: aws.openmcf.org/v1
kind: AwsKinesisFirehose
metadata:
  name: log-analytics-firehose
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.AwsKinesisFirehose.log-analytics-firehose
spec:
  region: us-east-1
  opensearch:
    domainArn:
      valueFrom:
        kind: AwsOpenSearchDomain
        name: my-log-domain
        fieldPath: status.outputs.domain_arn
    indexName: application-logs
    roleArn: arn:aws:iam::123456789012:role/firehose-opensearch-role
    indexRotationPeriod: OneDay
    s3BackupMode: FailedDocumentsOnly
    buffering:
      intervalInSeconds: 60
      sizeInMbs: 5
    s3Config:
      bucketArn: arn:aws:s3:::my-firehose-backup-bucket
      roleArn: arn:aws:iam::123456789012:role/firehose-s3-backup-role
      prefix: opensearch-backup/failed/
      compressionFormat: GZIP

Production S3 with Kinesis Source and Parquet Conversion

Consumes from an existing Kinesis Data Stream, converts JSON to Parquet via AWS Glue Data Catalog, and writes columnar files to a partitioned S3 data lake:

apiVersion: aws.openmcf.org/v1
kind: AwsKinesisFirehose
metadata:
  name: analytics-parquet-firehose
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.AwsKinesisFirehose.analytics-parquet-firehose
spec:
  region: us-east-1
  kinesisStreamSource:
    streamArn:
      valueFrom:
        kind: AwsKinesisStream
        name: my-events-stream
        fieldPath: status.outputs.stream_arn
    roleArn:
      valueFrom:
        kind: AwsIamRole
        name: firehose-kinesis-consumer-role
        fieldPath: status.outputs.role_arn
  extendedS3:
    bucketArn: arn:aws:s3:::my-analytics-data-lake
    roleArn: arn:aws:iam::123456789012:role/firehose-glue-s3-role
    prefix: analytics/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/
    compressionFormat: UNCOMPRESSED
    fileExtension: .parquet
    buffering:
      intervalInSeconds: 60
      sizeInMbs: 64
    dynamicPartitioning:
      enabled: true
      retryDurationInSeconds: 300
    dataFormatConversion:
      enabled: true
      inputFormat: OPENX_JSON
      outputFormat: PARQUET
      parquetCompression: SNAPPY
      schema:
        databaseName: analytics_db
        tableName: events
        roleArn: arn:aws:iam::123456789012:role/firehose-glue-access-role

Stack Outputs

After deployment, the following outputs are available in status.outputs:

Related Components

• AwsKinesisStream — provides a Kinesis Data Stream as the source for the delivery stream
• AwsS3Bucket — serves as the delivery destination, backup target, or Redshift staging area
• AwsOpenSearchDomain — serves as the indexing destination for log and search workloads
• AwsIamRole — provides the permissions Firehose needs for source, destination, and transformation access
• AwsLambda — provides the Lambda function for record transformation before delivery