Skip to content
~/sph.sh

SNS/SQS Cross-Account Fan-Out: Building Multi-Account Event Distribution in AWS

Learn how to implement secure cross-account event distribution using Amazon SNS and SQS. Covers IAM policies, KMS encryption, AWS CDK implementation, and common pitfalls from real-world deployments.

Abstract

Cross-account SNS/SQS fan-out enables secure event distribution across AWS account boundaries. This architecture pattern allows a single SNS topic in one account to deliver messages to multiple SQS queues in different accounts, maintaining administrative isolation while enabling event-driven communication. This guide covers the complete implementation including IAM policies, KMS encryption, AWS CDK setup, and troubleshooting common issues that emerge in production.

Why Cross-Account Fan-Out Matters

Working with multi-account AWS Organizations taught me that proper event distribution is essential for organizational scale. When you have separate accounts for different teams, services, or environments, you need a way to share events without compromising security boundaries.

The SNS/SQS fan-out pattern solves several real problems:

Administrative isolation: Each account maintains independent control over its resources. The billing team can't accidentally delete the fulfillment team's infrastructure, even though they both receive events from the same source.

Independent scaling: Consumer accounts scale their SQS processing independently. One slow consumer doesn't impact others - messages queue up in their account while others continue processing.

Cost efficiency: SNS to SQS delivery is free (you only pay for SNS publishes and SQS operations). Compared to HTTP endpoints or other integration methods, this saves significant costs at scale.

Security boundaries: Each account implements its own encryption, access policies, and compliance controls. The security team can enforce strict key management in their account without requiring changes to the publisher.

Architecture Overview

Here's how the cross-account fan-out pattern works:

The pattern requires proper configuration at three levels:

  1. SNS topic policy: Grants cross-account sns:Subscribe permission
  2. SQS queue policy: Allows SNS service principal to sqs:SendMessage
  3. KMS key policy (if encrypted): Permits SNS to encrypt/decrypt messages

IAM Policies and Permissions

Getting cross-account permissions right is critical. Here's what I've learned works reliably.

SNS Topic Policy (Publisher Account)

The SNS topic must explicitly grant sns:Subscribe permission to target accounts:

typescript
import * as sns from 'aws-cdk-lib/aws-sns';import * as iam from 'aws-cdk-lib/aws-iam';import { Stack, StackProps } from 'aws-cdk-lib';import { Construct } from 'constructs';
export class PublisherStack extends Stack {  public readonly topic: sns.Topic;
  constructor(scope: Construct, id: string, props?: StackProps) {    super(scope, id, props);
    // Create SNS topic    this.topic = new sns.Topic(this, 'CentralEventTopic', {      topicName: 'central-events',      displayName: 'Central Event Distribution Topic',    });
    // Grant cross-account subscribe permissions    this.topic.addToResourcePolicy(      new iam.PolicyStatement({        sid: 'AllowCrossAccountSubscribe',        effect: iam.Effect.ALLOW,        principals: [          new iam.AccountPrincipal('111111111111'), // Account A          new iam.AccountPrincipal('222222222222'), // Account B          new iam.AccountPrincipal('333333333333'), // Account C        ],        actions: ['sns:Subscribe'],        resources: [this.topic.topicArn],      })    );
    // Optionally allow specific IAM roles instead of entire accounts    // This is more restrictive and follows least-privilege principle    this.topic.addToResourcePolicy(      new iam.PolicyStatement({        sid: 'AllowSpecificRoleSubscribe',        effect: iam.Effect.ALLOW,        principals: [          new iam.ArnPrincipal('arn:aws:iam::111111111111:role/ServiceARole'),        ],        actions: ['sns:Subscribe'],        resources: [this.topic.topicArn],      })    );  }}

Key considerations:

  • Use AccountPrincipal for organization-wide access or ArnPrincipal for specific roles
  • The sns:Subscribe action is required for creating subscriptions
  • This policy doesn't grant message publishing - only subscription creation
  • You can add conditions to restrict by source VPC, IP range, or other factors

SQS Queue Policy (Consumer Account)

Each consumer account needs a queue policy allowing the SNS service principal to send messages:

typescript
import * as sqs from 'aws-cdk-lib/aws-sqs';import * as sns from 'aws-cdk-lib/aws-sns';import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';import * as iam from 'aws-cdk-lib/aws-iam';import { Duration, Stack, StackProps } from 'aws-cdk-lib';import { Construct } from 'constructs';
interface ConsumerStackProps extends StackProps {  centralTopicArn: string; // ARN from publisher account}
export class ConsumerStack extends Stack {  constructor(scope: Construct, id: string, props: ConsumerStackProps) {    super(scope, id, props);
    // Create dead letter queue for failed messages    const dlq = new sqs.Queue(this, 'EventDLQ', {      queueName: 'events-dlq',      retentionPeriod: Duration.days(14),    });
    // Create main event queue    const queue = new sqs.Queue(this, 'EventQueue', {      queueName: 'service-events',      visibilityTimeout: Duration.seconds(30),      receiveMessageWaitTime: Duration.seconds(20), // Enable long polling      deadLetterQueue: {        queue: dlq,        maxReceiveCount: 3,      },    });
    // Add queue policy allowing SNS to send messages    queue.addToResourcePolicy(      new iam.PolicyStatement({        sid: 'AllowSNSPublish',        effect: iam.Effect.ALLOW,        principals: [new iam.ServicePrincipal('sns.amazonaws.com')],        actions: ['sqs:SendMessage'],        resources: [queue.queueArn],        conditions: {          ArnEquals: {            'aws:SourceArn': props.centralTopicArn,          },        },      })    );
    // Import the cross-account SNS topic    const centralTopic = sns.Topic.fromTopicArn(      this,      'CentralTopic',      props.centralTopicArn    );
    // Subscribe queue to topic    centralTopic.addSubscription(      new subscriptions.SqsSubscription(queue, {        rawMessageDelivery: false, // Set to true to receive just the message body      })    );  }}

Important details:

  • The Condition with aws:SourceArn prevents other SNS topics from sending to your queue
  • rawMessageDelivery: false wraps the message in SNS metadata (recommended for debugging)
  • Set rawMessageDelivery: true if you only want the message body without SNS envelope
  • Long polling (receiveMessageWaitTime) reduces empty receives and costs

The Two-Way Handshake

Cross-account subscriptions require both accounts to agree:

  1. Publisher permits subscription: SNS topic policy grants sns:Subscribe to consumer account
  2. Consumer accepts messages: SQS queue policy allows SNS service principal to send messages
  3. Consumer creates subscription: Queue owner calls sns:Subscribe using the topic ARN

This two-way handshake is crucial. If either policy is missing, you'll get "Access Denied" errors. I've learned to always check both sides when troubleshooting subscription failures.

KMS Encryption Configuration

Encryption adds complexity to cross-account setups. AWS-managed keys don't work across account boundaries - you must use customer-managed keys.

Why AWS-Managed Keys Don't Work

When you create an SQS queue with encryption using the AWS-managed key (alias/aws/sqs), the key policy only grants permissions within that account. The SNS service in the publisher account can't use a consumer account's AWS-managed key.

Customer-Managed Key Setup

Here's a working pattern for encrypted queues:

typescript
import * as kms from 'aws-cdk-lib/aws-kms';import * as sqs from 'aws-cdk-lib/aws-sqs';import * as sns from 'aws-cdk-lib/aws-sns';import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';import * as iam from 'aws-cdk-lib/aws-iam';import * as cdk from 'aws-cdk-lib';import { Duration, Stack, StackProps } from 'aws-cdk-lib';import { Construct } from 'constructs';
interface EncryptedConsumerStackProps extends StackProps {  centralTopicArn: string;}
export class EncryptedConsumerStack extends Stack {  constructor(scope: Construct, id: string, props: EncryptedConsumerStackProps) {    super(scope, id, props);
    // Create customer-managed KMS key    const queueKey = new kms.Key(this, 'QueueEncryptionKey', {      description: 'KMS key for cross-account SQS queue encryption',      enableKeyRotation: true,      removalPolicy: cdk.RemovalPolicy.RETAIN, // Don't delete keys    });
    // Grant SNS service permission to use the key    queueKey.addToResourcePolicy(      new iam.PolicyStatement({        sid: 'AllowSNSToUseKey',        effect: iam.Effect.ALLOW,        principals: [new iam.ServicePrincipal('sns.amazonaws.com')],        actions: [          'kms:Decrypt',          'kms:GenerateDataKey',        ],        resources: ['*'],        conditions: {          StringEquals: {            // Ensure SNS uses this key only for SQS in this region            'kms:ViaService': `sqs.${this.region}.amazonaws.com`,          },        },      })    );
    // Create encrypted dead letter queue    const dlq = new sqs.Queue(this, 'EncryptedEventDLQ', {      queueName: 'encrypted-events-dlq',      encryptionMasterKey: queueKey,      retentionPeriod: Duration.days(14),    });
    // Create encrypted main queue    const queue = new sqs.Queue(this, 'EncryptedEventQueue', {      queueName: 'encrypted-service-events',      encryptionMasterKey: queueKey,      visibilityTimeout: Duration.seconds(30),      receiveMessageWaitTime: Duration.seconds(20),      deadLetterQueue: {        queue: dlq,        maxReceiveCount: 3,      },    });
    // Queue policy allowing SNS to send messages    queue.addToResourcePolicy(      new iam.PolicyStatement({        sid: 'AllowSNSPublish',        effect: iam.Effect.ALLOW,        principals: [new iam.ServicePrincipal('sns.amazonaws.com')],        actions: ['sqs:SendMessage'],        resources: [queue.queueArn],        conditions: {          ArnEquals: {            'aws:SourceArn': props.centralTopicArn,          },        },      })    );
    // Import cross-account topic and subscribe    const centralTopic = sns.Topic.fromTopicArn(      this,      'CentralTopic',      props.centralTopicArn    );
    centralTopic.addSubscription(      new subscriptions.SqsSubscription(queue)    );  }}

Key policy requirements:

  • kms:Decrypt: SNS needs this to decrypt messages when sending to the queue
  • kms:GenerateDataKey: Required for envelope encryption
  • kms:ViaService condition: Restricts key usage to SQS service in specific region
  • Enable key rotation for security best practices

Cost Consideration

Customer-managed KMS keys cost 1/monthperkey,plus1/month per key, plus 0.03 per 10,000 requests. For cross-account scenarios, this is required - there's no free alternative.

Message Filtering for Cost Optimization

SNS subscription filters reduce costs by preventing unwanted messages from reaching queues. Filtering happens at the SNS level before SQS charges apply.

Attribute-Based Filtering

Message attributes provide simple, efficient filtering:

typescript
// Publisher: Publish with attributesimport { SNSClient, PublishCommand } from '@aws-sdk/client-sns';
const sns = new SNSClient({ region: 'us-east-1' });
await sns.send(  new PublishCommand({    TopicArn: 'arn:aws:sns:us-east-1:999999999999:central-events',    Message: JSON.stringify({      orderId: '12345',      amount: 1500,      region: 'us-east-1',    }),    MessageAttributes: {      eventType: {        DataType: 'String',        StringValue: 'OrderCreated',      },      priority: {        DataType: 'String',        StringValue: 'high',      },      amount: {        DataType: 'Number',        StringValue: '1500',      },      region: {        DataType: 'String',        StringValue: 'us-east-1',      },    },  }));
typescript
// Consumer: Subscribe with filter policyimport * as sns from 'aws-cdk-lib/aws-sns';import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
centralTopic.addSubscription(  new subscriptions.SqsSubscription(highPriorityQueue, {    filterPolicy: {      // Only high priority events      priority: sns.SubscriptionFilter.stringFilter({        allowlist: ['high', 'critical'],      }),      // Only large orders      amount: sns.SubscriptionFilter.numericFilter({        greaterThan: 1000,      }),    },  }));
centralTopic.addSubscription(  new subscriptions.SqsSubscription(regionalQueue, {    filterPolicy: {      // Only specific regions      region: sns.SubscriptionFilter.stringFilter({        allowlist: ['us-east-1', 'eu-west-1'],      }),    },  }));
// Analytics queue receives everything (no filter)centralTopic.addSubscription(  new subscriptions.SqsSubscription(analyticsQueue));

Payload-Based Filtering

Newer payload-based filtering (introduced in 2024) allows filtering on the message body itself:

typescript
import { SNSClient, SubscribeCommand } from '@aws-sdk/client-sns';
const sns = new SNSClient({ region: 'us-east-1' });
await sns.send(  new SubscribeCommand({    TopicArn: 'arn:aws:sns:us-east-1:999999999999:central-events',    Protocol: 'sqs',    Endpoint: 'arn:aws:sqs:us-east-1:111111111111:service-events',    Attributes: {      FilterPolicyScope: 'MessageBody',      FilterPolicy: JSON.stringify({        order: {          status: ['completed', 'shipped'],          amount: [{ numeric: ['>', 1000] }],        },      }),    },  }));

Filter policy benefits:

  • Reduces SQS request costs by 50-90% in typical scenarios
  • Each subscriber receives only relevant messages
  • Filter changes take up to 15 minutes to propagate
  • Up to 200 filter policies per topic

FIFO Topics and Queues

FIFO (First-In-First-Out) topics provide strict ordering and exactly-once delivery. Use them when message order matters.

When to Use FIFO

FIFO makes sense for:

  • Order processing workflows where sequence matters
  • Financial transactions requiring exactly-once processing
  • State machine transitions that must occur in order
  • Inventory updates where order impacts final state

FIFO Setup Requirements

typescript
import * as sns from 'aws-cdk-lib/aws-sns';import * as sqs from 'aws-cdk-lib/aws-sqs';import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
// Create FIFO topic (publisher account)const fifoTopic = new sns.Topic(this, 'OrderEventTopic', {  topicName: 'order-events.fifo',  fifo: true,  contentBasedDeduplication: true,  // High-throughput mode: 3000+ TPS per message group  fifoThroughputScope: sns.FifoThroughputScope.MESSAGE_GROUP,});
// Create FIFO queue (consumer account)const fifoQueue = new sqs.Queue(this, 'OrderQueue', {  queueName: 'orders.fifo',  fifo: true,  contentBasedDeduplication: true,});
// FIFO topic can only subscribe FIFO queuesfifoTopic.addSubscription(  new subscriptions.SqsSubscription(fifoQueue));

Publishing to FIFO topics:

typescript
import { SNSClient, PublishCommand } from '@aws-sdk/client-sns';
const sns = new SNSClient({ region: 'us-east-1' });
await sns.send(  new PublishCommand({    TopicArn: 'arn:aws:sns:us-east-1:999999999999:order-events.fifo',    Message: JSON.stringify({ orderId: '12345', status: 'created' }),    MessageGroupId: 'order-region-us-east-1', // Required for ordering    MessageDeduplicationId: 'order-12345-created', // Optional if contentBasedDeduplication enabled  }));

High-throughput mode considerations:

  • Default FIFO: 300 TPS with topic-level deduplication
  • High-throughput mode: 30,000 TPS with message-group-level deduplication (increased January 2025)
  • Cannot be reversed once enabled
  • Use multiple message groups to parallelize processing

Common Pitfalls and Solutions

Here are issues I've encountered in production and their solutions.

Pitfall 1: Subscription Shows "PendingConfirmation"

Symptom: Subscription created but stuck in "PendingConfirmation" status. Messages never flow.

Root cause: When the topic owner creates the subscription (rather than the queue owner), SNS sends a confirmation message that must be manually confirmed.

Solution: Always have the queue owner create the subscription:

typescript
// PREFERRED: Queue owner subscribes (in consumer account)const centralTopic = sns.Topic.fromTopicArn(  this,  'CentralTopic',  'arn:aws:sns:us-east-1:999999999999:central-events');
centralTopic.addSubscription(  new subscriptions.SqsSubscription(queue));// No confirmation needed - subscription active immediately

If you must have the topic owner create subscriptions, automate confirmation:

python
# Python script to auto-confirm subscriptionsimport boto3import json
sqs = boto3.client('sqs')queue_url = 'https://sqs.us-east-1.amazonaws.com/111111111111/service-events'
# Poll for confirmation messageresponse = sqs.receive_message(    QueueUrl=queue_url,    MaxNumberOfMessages=1,    WaitTimeSeconds=10)
for message in response.get('Messages', []):    body = json.loads(message['Body'])
    if 'SubscribeURL' in body:        # Confirm subscription by visiting URL        import urllib.request        urllib.request.urlopen(body['SubscribeURL'])        print(f"Confirmed subscription: {body['SubscribeURL']}")
        # Delete confirmation message        sqs.delete_message(            QueueUrl=queue_url,            ReceiptHandle=message['ReceiptHandle']        )

Pitfall 2: KMS Key Access Denied

Symptom: Messages published to SNS but never appear in encrypted SQS queue. No errors in SNS metrics.

Root cause: SNS service lacks permission to use the KMS key for encryption.

Solution: Verify KMS key policy grants SNS the required permissions:

typescript
// Check your KMS key policy includes thisqueueKey.addToResourcePolicy(  new iam.PolicyStatement({    effect: iam.Effect.ALLOW,    principals: [new iam.ServicePrincipal('sns.amazonaws.com')],    actions: [      'kms:Decrypt',      'kms:GenerateDataKey',    ],    resources: ['*'],    conditions: {      StringEquals: {        'kms:ViaService': `sqs.${this.region}.amazonaws.com`,      },    },  }));

Troubleshooting tip: Check CloudTrail logs for KMS AccessDenied errors:

bash
aws cloudtrail lookup-events \  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \  --max-results 50 \  --region us-east-1 \  --query 'Events[?ErrorCode==`AccessDenied`]'

Pitfall 3: Region Mismatch

Symptom: "Access Denied" errors despite correct policies.

Root cause: SNS topic and SQS queue in different regions. Cross-region direct subscriptions aren't supported for cross-account scenarios.

Solution: Keep SNS topic and SQS queues in the same region. For multi-region requirements, use SNS to Lambda forwarders:

typescript
// Region 1 (us-east-1): Original topicconst sourceTopicEast = new sns.Topic(this, 'SourceTopicEast', {  topicName: 'events-us-east-1',});
// Lambda forwarder publishes to region 2 topicconst forwarder = new lambda.Function(this, 'RegionForwarder', {  runtime: lambda.Runtime.NODEJS_20_X,  handler: 'index.handler',  code: lambda.Code.fromInline(`    const { SNSClient, PublishCommand } = require('@aws-sdk/client-sns');    const sns = new SNSClient({ region: 'eu-west-1' });
    exports.handler = async (event) => {      for (const record of event.Records) {        await sns.send(new PublishCommand({          TopicArn: process.env.TARGET_TOPIC_ARN,          Message: record.Sns.Message,          MessageAttributes: record.Sns.MessageAttributes,        }));      }    };  `),  environment: {    TARGET_TOPIC_ARN: 'arn:aws:sns:eu-west-1:999999999999:events-eu-west-1',  },});
sourceTopicEast.addSubscription(  new subscriptions.LambdaSubscription(forwarder));

Pitfall 4: Message Size Limits

Symptom: Some messages delivered successfully, others silently disappear.

Root cause: SNS and SQS both have 256 KB message size limits. Messages exceeding this are dropped without notification.

Solution: Keep messages under 256 KB or use S3 for large payloads:

typescript
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';import { SNSClient, PublishCommand } from '@aws-sdk/client-sns';
const s3 = new S3Client({ region: 'us-east-1' });const sns = new SNSClient({ region: 'us-east-1' });
async function publishLargeMessage(topicArn: string, payload: any) {  const payloadSize = Buffer.byteLength(JSON.stringify(payload));
  if (payloadSize > 200_000) {    // Store large payload in S3    const messageId = crypto.randomUUID();    const s3Key = `messages/${messageId}.json`;
    await s3.send(      new PutObjectCommand({        Bucket: 'large-message-payloads',        Key: s3Key,        Body: JSON.stringify(payload),      })    );
    // Publish reference to S3 object    await sns.send(      new PublishCommand({        TopicArn: topicArn,        Message: JSON.stringify({          type: 'S3Reference',          bucket: 'large-message-payloads',          key: s3Key,        }),      })    );  } else {    // Direct publish for small messages    await sns.send(      new PublishCommand({        TopicArn: topicArn,        Message: JSON.stringify(payload),      })    );  }}

Pitfall 5: Filter Policies Not Taking Effect

Symptom: Messages still delivered despite filter policy.

Root cause: Filter policies take up to 15 minutes to propagate, or message attributes don't match filter format.

Solution: Wait for propagation and verify attribute format:

typescript
// Verify message attributes match filter expectationsawait sns.send(  new PublishCommand({    TopicArn: topicArn,    Message: JSON.stringify({ orderId: '12345' }),    MessageAttributes: {      eventType: {        DataType: 'String',        StringValue: 'OrderCreated', // Must match filter exactly      },      priority: {        DataType: 'Number', // Use Number type for numeric filters        StringValue: '5',      },    },  }));
// Monitor filtering effectivenessconst filterMetric = new cloudwatch.Metric({  namespace: 'AWS/SNS',  metricName: 'NumberOfNotificationsFilteredOut',  dimensionsMap: {    TopicName: 'central-events',  },});

Monitoring and Observability

Effective monitoring is essential for cross-account messaging. You need visibility into both publisher and consumer sides.

Key SNS Metrics

typescript
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';import * as cloudwatch_actions from 'aws-cdk-lib/aws-cloudwatch-actions';import * as sns from 'aws-cdk-lib/aws-sns';
// Monitor SNS delivery failuresnew cloudwatch.Alarm(this, 'SNSDeliveryFailures', {  metric: topic.metricNumberOfNotificationsFailed({    statistic: 'Sum',    period: Duration.minutes(5),  }),  threshold: 10,  evaluationPeriods: 2,  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,  alarmDescription: 'Alert when SNS message delivery fails',});
// Monitor filter effectivenessnew cloudwatch.Alarm(this, 'HighFilterRate', {  metric: topic.metricNumberOfNotificationsFilteredOut({    statistic: 'Sum',    period: Duration.minutes(5),  }),  threshold: 1000,  evaluationPeriods: 1,  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,  alarmDescription: 'Alert when filter rate is unusually high',});

Key SQS Metrics

typescript
// Monitor queue depthnew cloudwatch.Alarm(this, 'QueueDepthAlarm', {  metric: queue.metricApproximateNumberOfMessagesVisible({    statistic: 'Average',    period: Duration.minutes(5),  }),  threshold: 1000,  evaluationPeriods: 2,  alarmDescription: 'Alert when queue depth grows too large',});
// Monitor processing lagnew cloudwatch.Alarm(this, 'OldMessageAlarm', {  metric: queue.metricApproximateAgeOfOldestMessage({    statistic: 'Maximum',    period: Duration.minutes(1),  }),  threshold: 300, // 5 minutes  evaluationPeriods: 3,  alarmDescription: 'Alert when messages are not processed timely',});
// Monitor DLQnew cloudwatch.Alarm(this, 'DLQMessages', {  metric: dlq.metricApproximateNumberOfMessagesVisible({    statistic: 'Sum',    period: Duration.minutes(5),  }),  threshold: 1,  evaluationPeriods: 1,  alarmDescription: 'Alert on any messages in DLQ',});

Cross-Account CloudWatch Observability

For unified monitoring across accounts, use CloudWatch Observability Access Manager:

typescript
import * as oam from 'aws-cdk-lib/aws-oam';
// In monitoring account: Create sinkconst sink = new oam.CfnSink(this, 'MonitoringSink', {  name: 'central-monitoring-sink',  policy: {    Version: '2012-10-17',    Statement: [      {        Effect: 'Allow',        Principal: {          AWS: [            'arn:aws:iam::111111111111:root',            'arn:aws:iam::222222222222:root',            'arn:aws:iam::333333333333:root',          ],        },        Action: ['oam:CreateLink', 'oam:UpdateLink'],        Resource: '*',      },    ],  },});
// In each source account: Create link to sinkconst link = new oam.CfnLink(this, 'MonitoringLink', {  resourceTypes: ['AWS::CloudWatch::Metric', 'AWS::Logs::LogGroup'],  sinkIdentifier: 'arn:aws:oam:us-east-1:999999999999:sink/sink-id',});

This enables a single dashboard showing metrics from all accounts without data transfer costs (within same region).

Cost Analysis

Understanding costs helps optimize your architecture.

Pricing Breakdown (2025)

SNS costs:

  • First 1 million requests/month: FREE
  • Beyond free tier: $0.50 per million publishes
  • SNS to SQS deliveries: FREE (major cost advantage)

SQS costs:

  • First 1 million requests/month: FREE
  • Standard queue: $0.40 per million requests
  • FIFO queue: $0.50 per million requests
  • Each 64 KB chunk = 1 request (256 KB message = 4 requests)

Fan-out cost example (1 million messages to 4 queues):

  • SNS publishes: 1M × 0.50=0.50 = 0.50
  • SNS to SQS delivery: FREE
  • SQS receives: 4M × 0.40=0.40 = 1.60
  • SQS deletes: 4M × 0.40=0.40 = 1.60
  • Total: $3.70

With 50% message filtering:

  • SNS publishes: 1M × 0.50=0.50 = 0.50
  • Filtered deliveries: 2M messages delivered
  • SQS receives: 2M × 0.40=0.40 = 0.80
  • SQS deletes: 2M × 0.40=0.40 = 0.80
  • Total: $2.10 (43% cost reduction)

KMS costs (for encrypted queues):

  • Customer-managed key: $1/month per key
  • KMS requests: $0.03 per 10,000 requests
  • Each encrypted message generates 2 KMS requests

Cost Optimization Strategies

  1. Implement message filtering: 50-70% cost reduction in typical scenarios
  2. Enable SQS long polling: Reduces empty receives by 90%
  3. Use batch operations: Up to 10× reduction in API calls
  4. Keep messages under 64 KB: Avoid multi-request charges
  5. Use Standard queues when ordering isn't critical: 20% cheaper than FIFO

Alternative Approaches

SNS/SQS fan-out isn't always the best choice. Here are alternatives and when to consider them.

EventBridge

When to use:

  • Need complex event routing (100+ rules)
  • Schema registry and validation required
  • Event replay capability essential
  • Integration with 30+ AWS services

Trade-offs:

  • 1.00permillionevents(vsSNS1.00 per million events (vs SNS 0.50)
  • More powerful filtering with JSONPath-like syntax
  • Built-in schema discovery and validation
  • Native cross-account event buses
typescript
import * as events from 'aws-cdk-lib/aws-events';import * as targets from 'aws-cdk-lib/aws-events-targets';
const bus = new events.EventBus(this, 'CentralBus', {  eventBusName: 'central-events',});
// Grant cross-account accessbus.grantPutEventsTo(new iam.AccountPrincipal('111111111111'));
// Complex filteringnew events.Rule(this, 'HighValueOrders', {  eventBus: bus,  eventPattern: {    source: ['com.myapp.orders'],    detailType: ['OrderCreated'],    detail: {      amount: [{ numeric: ['>', 1000] }],      region: ['us-east-1', 'us-west-2'],      status: ['pending', 'processing'],    },  },  targets: [new targets.SqsQueue(queue)],});

Kinesis Data Streams

When to use:

  • Ordered stream processing required
  • Need replay capability (up to 365 days)
  • Multiple consumers reading at different speeds
  • Real-time analytics use cases

Trade-offs:

  • More expensive ($0.015 per shard hour + PUT costs)
  • Complex shard management
  • Better for streaming analytics than discrete events
  • Higher operational overhead

Direct Lambda Invocation

When to use:

  • Synchronous processing acceptable
  • Event volume under Lambda concurrent execution limits
  • No need for queue management
  • Simple, fast processing logic

Trade-offs:

  • No built-in retry queues
  • Cold start considerations
  • Limited by Lambda concurrency
  • Less flexible than queues for scaling

Real-World Implementation Pattern

Here's a complete, production-ready multi-account setup:

typescript
// publisher-stack.tsimport * as cdk from 'aws-cdk-lib';import * as sns from 'aws-cdk-lib/aws-sns';import * as iam from 'aws-cdk-lib/aws-iam';import { Construct } from 'constructs';
export class PublisherStack extends cdk.Stack {  public readonly topicArn: string;
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {    super(scope, id, props);
    const topic = new sns.Topic(this, 'CentralEvents', {      topicName: 'central-events',      displayName: 'Central Event Distribution',    });
    // Allow multiple consumer accounts    topic.addToResourcePolicy(      new iam.PolicyStatement({        sid: 'AllowConsumerSubscribe',        principals: [          new iam.AccountPrincipal('111111111111'),          new iam.AccountPrincipal('222222222222'),          new iam.AccountPrincipal('333333333333'),        ],        actions: ['sns:Subscribe'],        resources: [topic.topicArn],      })    );
    this.topicArn = topic.topicArn;
    // Output for cross-stack references    new cdk.CfnOutput(this, 'TopicArnOutput', {      value: topic.topicArn,      exportName: 'CentralEventTopicArn',    });  }}
// consumer-stack.tsimport * as cdk from 'aws-cdk-lib';import * as sqs from 'aws-cdk-lib/aws-sqs';import * as sns from 'aws-cdk-lib/aws-sns';import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';import * as lambda from 'aws-cdk-lib/aws-lambda';import * as lambdaEventSources from 'aws-cdk-lib/aws-lambda-event-sources';import * as iam from 'aws-cdk-lib/aws-iam';import * as kms from 'aws-cdk-lib/aws-kms';import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';import { Construct } from 'constructs';
interface ConsumerStackProps extends cdk.StackProps {  centralTopicArn: string;  serviceName: string;}
export class ConsumerStack extends cdk.Stack {  constructor(scope: Construct, id: string, props: ConsumerStackProps) {    super(scope, id, props);
    // Create KMS key for encryption    const encryptionKey = new kms.Key(this, 'EncryptionKey', {      description: `Encryption key for ${props.serviceName} events`,      enableKeyRotation: true,    });
    // Grant SNS permission to use key    encryptionKey.addToResourcePolicy(      new iam.PolicyStatement({        principals: [new iam.ServicePrincipal('sns.amazonaws.com')],        actions: ['kms:Decrypt', 'kms:GenerateDataKey'],        resources: ['*'],        conditions: {          StringEquals: {            'kms:ViaService': `sqs.${this.region}.amazonaws.com`,          },        },      })    );
    // Create DLQ    const dlq = new sqs.Queue(this, 'DLQ', {      queueName: `${props.serviceName}-dlq`,      encryptionMasterKey: encryptionKey,      retentionPeriod: cdk.Duration.days(14),    });
    // Create main queue    const queue = new sqs.Queue(this, 'EventQueue', {      queueName: `${props.serviceName}-events`,      encryptionMasterKey: encryptionKey,      visibilityTimeout: cdk.Duration.seconds(30),      receiveMessageWaitTime: cdk.Duration.seconds(20),      deadLetterQueue: {        queue: dlq,        maxReceiveCount: 3,      },    });
    // Allow SNS to send messages    queue.addToResourcePolicy(      new iam.PolicyStatement({        principals: [new iam.ServicePrincipal('sns.amazonaws.com')],        actions: ['sqs:SendMessage'],        resources: [queue.queueArn],        conditions: {          ArnEquals: {            'aws:SourceArn': props.centralTopicArn,          },        },      })    );
    // Subscribe to central topic    const centralTopic = sns.Topic.fromTopicArn(      this,      'CentralTopic',      props.centralTopicArn    );
    centralTopic.addSubscription(      new subscriptions.SqsSubscription(queue, {        rawMessageDelivery: false,      })    );
    // Create processor Lambda    const processor = new lambda.Function(this, 'EventProcessor', {      runtime: lambda.Runtime.NODEJS_20_X,      handler: 'index.handler',      code: lambda.Code.fromInline(`        exports.handler = async (event) => {          for (const record of event.Records) {            const snsMessage = JSON.parse(record.body);            const message = JSON.parse(snsMessage.Message);
            console.log('Processing message:', message);
            // Your business logic here
            // Message automatically deleted if handler succeeds          }        };      `),      timeout: cdk.Duration.seconds(30),      environment: {        SERVICE_NAME: props.serviceName,      },    });
    // Connect queue to Lambda    processor.addEventSource(      new lambdaEventSources.SqsEventSource(queue, {        batchSize: 10,        reportBatchItemFailures: true,      })    );
    // CloudWatch alarms    new cloudwatch.Alarm(this, 'QueueDepthAlarm', {      metric: queue.metricApproximateNumberOfMessagesVisible(),      threshold: 1000,      evaluationPeriods: 2,    });
    new cloudwatch.Alarm(this, 'DLQMessagesAlarm', {      metric: dlq.metricApproximateNumberOfMessagesVisible(),      threshold: 1,      evaluationPeriods: 1,    });
    new cloudwatch.Alarm(this, 'ProcessorErrorsAlarm', {      metric: processor.metricErrors(),      threshold: 10,      evaluationPeriods: 2,    });  }}

Key Takeaways

Working with cross-account SNS/SQS taught me several important lessons:

Always use customer-managed KMS keys for encrypted cross-account queues. AWS-managed keys simply don't work across account boundaries. The $1/month per key is unavoidable.

Have the queue owner create subscriptions. This eliminates the manual confirmation step and reduces setup complexity. When the topic owner creates subscriptions, you need automation to handle confirmation messages.

Implement comprehensive monitoring from the start. Cross-account troubleshooting is significantly harder without proper CloudWatch metrics. Set up alarms in both publisher and consumer accounts.

Filter at the SNS level with subscription filters. This reduces costs by 50-90% in typical scenarios. Filtering happens before SQS charges apply, making it highly cost-effective.

Keep SNS topics and SQS queues in the same region. Cross-region subscriptions add significant complexity. If you need multi-region distribution, use Lambda forwarders.

DLQs must be in the subscriber account. You can't use a DLQ from the publisher account for cross-account subscriptions. Each consumer account needs its own DLQ.

Plan for 15-minute filter policy propagation. Don't expect immediate changes when updating filter policies. Test filter changes in non-production first.

The SNS/SQS fan-out pattern provides reliable, cost-effective event distribution across AWS accounts. When you need persistent queues with independent consumer scaling and strong administrative boundaries, this architecture delivers excellent results. The implementation complexity is manageable once you understand the permission model and encryption requirements.

References

aws
aws-sns
aws-sqs
event-driven
serverless
aws-cdk
typescript
security
architecture

Related Posts