OpenMCF logoOpenMCF

Loading...

AWS OpenSearch Domain

Deploys an Amazon OpenSearch Service domain with configurable cluster topology, EBS storage, encryption at rest and in transit, optional VPC placement, fine-grained access control, UltraWarm and cold storage tiers, log publishing to CloudWatch, and Auto-Tune optimization.

What Gets Created

When you deploy an AwsOpenSearchDomain resource, OpenMCF provisions:

  • OpenSearch Domain — an opensearch.Domain with the specified engine version, cluster configuration, and tags derived from metadata (organization, environment, resource kind, resource ID)
  • EBS Volumes — attached to each data node for index storage, with configurable volume type, size, IOPS, and throughput
  • Encryption at Rest — optional KMS-based encryption of indices and automated snapshots, using the AWS-managed aws/es key or a customer-managed key
  • Node-to-Node Encryption — optional TLS encryption for all inter-node traffic within the cluster
  • VPC Endpoints (when vpcOptions is set) — ENIs deployed into specified subnets with security groups controlling inbound/outbound traffic on port 443; the domain is not publicly accessible
  • Fine-Grained Access Control (when advancedSecurityOptions.enabled is true) — internal user database or IAM-based authentication with role-based index-level permissions
  • Log Publishing (when logPublishingOptions is set) — CloudWatch Logs delivery for index slow logs, search slow logs, application logs, and audit logs
  • Auto-Tune (when autoTuneEnabled is true) — automatic JVM heap, disk I/O, and performance setting optimization based on cluster metrics

Prerequisites

  • AWS credentials configured via environment variables or OpenMCF provider config
  • An OpenSearch or Elasticsearch engine version string (e.g., OpenSearch_2.11 or Elasticsearch_7.10)
  • (Optional) VPC subnets and security groups for VPC-based deployment
  • (Optional) A KMS key ARN for customer-managed encryption at rest
  • (Optional) CloudWatch Logs log group ARNs for log publishing
  • (Optional) An ACM certificate ARN for custom domain endpoints

Quick Start

Create a file opensearch-domain.yaml:

apiVersion: aws.openmcf.org/v1
kind: AwsOpenSearchDomain
metadata:
  name: my-search
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: dev.AwsOpenSearchDomain.my-search
spec:
  region: us-west-2
  engineVersion: "OpenSearch_2.11"
  clusterConfig:
    instanceType: t3.small.search
  ebsOptions:
    ebsEnabled: true
    volumeType: gp3
    volumeSize: 20
  encryptAtRestEnabled: true
  nodeToNodeEncryptionEnabled: true

Deploy:

openmcf apply -f opensearch-domain.yaml

This creates a single-node OpenSearch 2.11 domain with 20 GB gp3 storage, encryption at rest, and node-to-node encryption.

Configuration Reference

Required Fields

FieldTypeDescriptionValidation
regionstringAWS region where the OpenSearch domain will be created (e.g., us-west-2, eu-west-1).Required; non-empty
engineVersionstringEngine version. Format: OpenSearch_X.Y or Elasticsearch_X.Y. Changing to an incompatible version forces domain recreation.Must match OpenSearch_X.Y or Elasticsearch_X.Y.
clusterConfigobjectCluster topology configuration.Required.
clusterConfig.instanceTypestringInstance type for data nodes. Uses the .search suffix.Must be set. Examples: t3.small.search, r6g.large.search.
ebsOptionsobjectEBS volume configuration for data node storage.Required.

Optional Fields

FieldTypeDefaultDescription
clusterConfig.instanceCountint321Number of data node instances. For zone-aware deployments, use a multiple of the availability zone count.
clusterConfig.dedicatedMasterEnabledboolfalseEnable dedicated master nodes for cluster management. Recommended for production.
clusterConfig.dedicatedMasterTypestring—Instance type for dedicated master nodes. Only used when dedicatedMasterEnabled is true.
clusterConfig.dedicatedMasterCountint32—Number of dedicated master nodes. AWS recommends 3 for production. Only used when dedicatedMasterEnabled is true.
clusterConfig.zoneAwarenessEnabledboolfalseDistribute data nodes and replicas across multiple Availability Zones.
clusterConfig.availabilityZoneCountint32—Number of AZs (2 or 3). Only used when zoneAwarenessEnabled is true.
clusterConfig.warmEnabledboolfalseEnable UltraWarm storage tier for infrequently accessed, read-only data.
clusterConfig.warmTypestring—Instance type for UltraWarm nodes (e.g., ultrawarm1.medium.search). Only used when warmEnabled is true.
clusterConfig.warmCountint32—Number of UltraWarm nodes (2–150). Only used when warmEnabled is true.
clusterConfig.coldStorageEnabledboolfalseEnable cold storage backed by S3. Requires warmEnabled to be true.
clusterConfig.multiAzWithStandbyEnabledboolfalseEnable Multi-AZ with Standby for 99.99% SLA. Requires 3 AZs and at least 3 data nodes.
ebsOptions.ebsEnabledboolfalseAttach EBS volumes to data nodes. Required for most instance types.
ebsOptions.volumeTypestring—EBS volume type: gp3 (recommended), gp2, io1, or standard.
ebsOptions.volumeSizeint32—Size of each EBS volume in GB. Total storage = volumeSize × instanceCount.
ebsOptions.iopsint32—Provisioned IOPS. Only valid for gp3 and io1. gp3 baseline: 3000.
ebsOptions.throughputint32—Provisioned throughput in MiB/s. Only valid for gp3. Minimum: 125.
encryptAtRestEnabledboolfalseEnable encryption at rest for indices and automated snapshots.
kmsKeyIdstring—Customer-managed KMS key ARN or ID for at-rest encryption. ForceNew. Can reference AwsKmsKey via valueFrom.
nodeToNodeEncryptionEnabledboolfalseEnable TLS encryption for all inter-node traffic.
vpcOptions.subnetIdsstring[][]Subnet IDs where OpenSearch deploys ENIs. For zone-aware domains, provide subnets in 2 or 3 AZs. ForceNew. Can reference AwsVpc via valueFrom.
vpcOptions.securityGroupIdsstring[][]Security group IDs controlling traffic to the domain. Must allow HTTPS (port 443). Can reference AwsSecurityGroup via valueFrom.
domainEndpointOptions.enforceHttpsbooltrueRequire HTTPS for all traffic to the domain endpoint.
domainEndpointOptions.tlsSecurityPolicystring—TLS policy for the HTTPS endpoint (e.g., Policy-Min-TLS-1-2-PFS-2023-10).
domainEndpointOptions.customEndpointEnabledboolfalseEnable a custom domain endpoint instead of the AWS-generated endpoint.
domainEndpointOptions.customEndpointstring—FQDN for the custom endpoint (e.g., search.example.com). Only used when customEndpointEnabled is true.
domainEndpointOptions.customEndpointCertificateArnstring—ACM certificate ARN for the custom endpoint. Can reference AwsCertManagerCert via valueFrom.
advancedSecurityOptions.enabledboolfalseEnable fine-grained access control. ForceNew if disabling.
advancedSecurityOptions.internalUserDatabaseEnabledboolfalseEnable the internal user database for managing users and roles in OpenSearch Dashboards.
advancedSecurityOptions.masterUserArnstring—IAM entity ARN designated as master user. Mutually exclusive with masterUserName. Can reference via valueFrom.
advancedSecurityOptions.masterUserNamestring—Username for internal user database master user. Mutually exclusive with masterUserArn.
advancedSecurityOptions.masterUserPasswordstring—Password for internal user database master user. Must be 8+ characters with uppercase, lowercase, digit, and special character. Can reference via valueFrom.
logPublishingOptionslist[]Up to 4 log publishing configurations, one per log type.
logPublishingOptions[].logTypestring—Log type: INDEX_SLOW_LOGS, SEARCH_SLOW_LOGS, ES_APPLICATION_LOGS, or AUDIT_LOGS.
logPublishingOptions[].cloudwatchLogGroupArnstring—CloudWatch Logs log group ARN. Can reference via valueFrom.
logPublishingOptions[].enabledbooltrueWhether this log publishing option is active.
accessPoliciesobject—IAM-based access policy document (JSON object). Controls who can access the domain and its indices.
autoTuneEnabledboolfalseEnable Auto-Tune to optimize JVM heap, disk I/O, and other settings.
autoSoftwareUpdateEnabledboolfalseEnable automatic service software updates during the off-peak window.
ipAddressTypestringipv4IP address type: ipv4 or dualstack. Changing from dualstack to ipv4 forces recreation.
advancedOptionsmap<string,string>{}Low-level key-value configuration options (e.g., rest.action.multi.allow_explicit_index).

Examples

Production VPC Deployment with Dedicated Masters

A multi-AZ domain deployed into a VPC with dedicated master nodes, zone awareness across 3 AZs, and encryption enabled:

apiVersion: aws.openmcf.org/v1
kind: AwsOpenSearchDomain
metadata:
  name: prod-search
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.AwsOpenSearchDomain.prod-search
spec:
  region: us-east-1
  engineVersion: "OpenSearch_2.11"
  clusterConfig:
    instanceType: r6g.large.search
    instanceCount: 3
    dedicatedMasterEnabled: true
    dedicatedMasterType: r6g.large.search
    dedicatedMasterCount: 3
    zoneAwarenessEnabled: true
    availabilityZoneCount: 3
  ebsOptions:
    ebsEnabled: true
    volumeType: gp3
    volumeSize: 100
    iops: 3000
    throughput: 125
  encryptAtRestEnabled: true
  nodeToNodeEncryptionEnabled: true
  vpcOptions:
    subnetIds:
      - subnet-az1
      - subnet-az2
      - subnet-az3
    securityGroupIds:
      - sg-opensearch
  domainEndpointOptions:
    enforceHttps: true
    tlsSecurityPolicy: Policy-Min-TLS-1-2-PFS-2023-10
  autoTuneEnabled: true

Fine-Grained Access Control with Internal User Database

A domain with FGAC enabled using the internal user database for authentication:

apiVersion: aws.openmcf.org/v1
kind: AwsOpenSearchDomain
metadata:
  name: fgac-search
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: staging.AwsOpenSearchDomain.fgac-search
spec:
  region: us-east-1
  engineVersion: "OpenSearch_2.11"
  clusterConfig:
    instanceType: r6g.large.search
    instanceCount: 2
  ebsOptions:
    ebsEnabled: true
    volumeType: gp3
    volumeSize: 50
  encryptAtRestEnabled: true
  nodeToNodeEncryptionEnabled: true
  advancedSecurityOptions:
    enabled: true
    internalUserDatabaseEnabled: true
    masterUserName: admin
    masterUserPassword: "MyStr0ng!Pass"
  domainEndpointOptions:
    enforceHttps: true

Analytics Domain with UltraWarm and Cold Storage

A domain with hot, warm, and cold storage tiers for cost-effective log analytics:

apiVersion: aws.openmcf.org/v1
kind: AwsOpenSearchDomain
metadata:
  name: analytics-search
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.AwsOpenSearchDomain.analytics-search
spec:
  region: us-east-1
  engineVersion: "OpenSearch_2.11"
  clusterConfig:
    instanceType: r6g.xlarge.search
    instanceCount: 3
    dedicatedMasterEnabled: true
    dedicatedMasterType: r6g.large.search
    dedicatedMasterCount: 3
    zoneAwarenessEnabled: true
    availabilityZoneCount: 3
    warmEnabled: true
    warmType: ultrawarm1.medium.search
    warmCount: 2
    coldStorageEnabled: true
  ebsOptions:
    ebsEnabled: true
    volumeType: gp3
    volumeSize: 200
    iops: 6000
    throughput: 250
  encryptAtRestEnabled: true
  kmsKeyId: arn:aws:kms:us-east-1:123456789012:key/abcd-1234-5678
  nodeToNodeEncryptionEnabled: true
  autoTuneEnabled: true
  autoSoftwareUpdateEnabled: true

Using Foreign Key References

Reference other OpenMCF-managed resources instead of hardcoding IDs:

apiVersion: aws.openmcf.org/v1
kind: AwsOpenSearchDomain
metadata:
  name: ref-search
  labels:
    openmcf.org/provisioner: pulumi
    pulumi.openmcf.org/organization: my-org
    pulumi.openmcf.org/project: my-project
    pulumi.openmcf.org/stack.name: prod.AwsOpenSearchDomain.ref-search
spec:
  region: us-west-2
  engineVersion: "OpenSearch_2.11"
  clusterConfig:
    instanceType: r6g.large.search
    instanceCount: 2
    zoneAwarenessEnabled: true
    availabilityZoneCount: 2
  ebsOptions:
    ebsEnabled: true
    volumeType: gp3
    volumeSize: 80
  encryptAtRestEnabled: true
  kmsKeyId:
    valueFrom:
      kind: AwsKmsKey
      name: search-key
      field: status.outputs.key_arn
  nodeToNodeEncryptionEnabled: true
  vpcOptions:
    subnetIds:
      - valueFrom:
          kind: AwsVpc
          name: main-vpc
          field: status.outputs.private_subnets[0].id
      - valueFrom:
          kind: AwsVpc
          name: main-vpc
          field: status.outputs.private_subnets[1].id
    securityGroupIds:
      - valueFrom:
          kind: AwsSecurityGroup
          name: opensearch-sg
          field: status.outputs.security_group_id
  domainEndpointOptions:
    enforceHttps: true
    customEndpointEnabled: true
    customEndpoint: search.example.com
    customEndpointCertificateArn:
      valueFrom:
        kind: AwsCertManagerCert
        name: search-cert
        field: status.outputs.certificate_arn

Stack Outputs

After deployment, the following outputs are available in status.outputs:

OutputTypeDescription
domain_idstringUnique identifier assigned to the domain by AWS
domain_namestringName of the domain (matches the ID derived from metadata)
domain_arnstringAmazon Resource Name of the domain, used in IAM policies and cross-service permissions
endpointstringDomain endpoint for index, search, and data upload requests. For VPC domains this is a VPC endpoint; for public domains it is internet-accessible
dashboard_endpointstringOpenSearch Dashboards UI endpoint (endpoint/_dashboards)

Related Components

  • AwsVpc — provides subnets for VPC deployment
  • AwsSecurityGroup — controls network access to the domain
  • AwsKmsKey — provides KMS keys for at-rest encryption
  • AwsCertManagerCert — provides ACM certificates for custom endpoints
  • AwsRoute53Zone — hosts DNS zones for CNAME records pointing to the domain

Next article

AWS RDS Cluster

AWS RDS Cluster Deploys an Amazon Aurora DB cluster (MySQL or PostgreSQL) with automatic subnet group creation, managed security group configuration, optional Secrets Manager password management, and optional Serverless v2 scaling. The component handles cluster-level configuration; instance-level resources are managed separately. What Gets Created When you deploy an AwsRdsCluster resource, OpenMCF provisions: RDS Aurora Cluster — an rds.Cluster with the specified engine (aurora-mysql or...
Read next article
Presets
3 ready-to-deploy configurationsView presets →