Building a Multi-Region app with AWS CDK — Part 1

5 min readFeb 7, 2021

There are lots of tutorials of deploying basic CRUD applications with AWS CDK using API Gateway, and Lambda connecting to a DynamoDB table. The defacto starter set for a serverless application. You build and deploy, modify the hello-world application to match your needs, you’re happy! The API is running and acting perfectly correctly in your default region.. us-east-1…

The Issues Begin

Then November 25th, 2020 strikes. The whole region is down for hours, your API is inaccessible, people are angry at YOU for this inconvenience. You read more about us-east-1 and realize that it tends to have a history of outages… and plan to make the jump to another region.

December rolls around and you’ve successfully migrated your application to us-west-2. It wasn’t too difficult, your CDK app was able to tear down everything in us-east-1, and then deploy everything again in the new region. You let out the breath you’d been holding.

January 7th, 2021 started off really well, and as you started looking forward to the end of the day… the alerts start ringing. Your new region, us-west-2, has begun to act up! After two hours of sweating, you are back up.

Multi-Region Solution

After all this pain and stress, you’re determined not to be caught flat-footed again. While the goal of a multi-region setup seemed daunting, you roll up your sleeves and dive into it.

For our CRUD application, there are two components we need to deal with:

Ensuring our data is consistent between the regions.
That DNS always resolves to a region that is up.

Thankfully there are solutions to both of these issues in our existing tech stack.

Data

Single Region Solution: DynamoDB Table

Multi-Region Solution: DynamoDB Global Table

DNS

Single Region Solution: Route53 Simple Routing Policy

Multi-Region Solution: Route53 Latency Routing Policy

In part 1, I’ll dive into the complexities of implementing a multi-region data configuration.

DynamoDB Global Tables

The DynamoDB Global Tables are a managed solutions from AWS where they keep replicate data changes from one DynamoDB table to all related tables in other regions.

For simplicity of use, there can be a number of “gotchas” from an Infrastructure as Code perspective.

CDK Primary Region

While DynamoDB Global Tables are multi-master, multi-region, from an AWS CDK point of view, we deploy them in only a single region. The resource is configured to replicate to the other regions that you’re interested in.

So I would instantiate my stack in us-west-2, and configure it to replicate to us-east-1, us-east-2, and us-west-1.

import * as dynamodb from '@aws-cdk/aws-dynamodb'; 
// Stack was called in us-west-2 
const gloablTable = new dynamodb.Table(this, 'globalTable', {  
  partitionKey: { 
    name: 'id', 
    type: dynamodb.AttributeType.STRING 
  }, 
  billingMode: dynamodb.BillingMode.PROVISIONED, 
  replicationRegions: ['us-east-1', 'us-east-2', 'us-west-1'],
  removalPolicy: cdk.RemovalPolicy.DESTROY, }
);...

Since we want to reference our global table’s name later on as environment variables for our Lambdas, we will want to build this out into two stacks.

This will make our app resemble the following:

const appRegions = ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2']; const app = new cdk.App(); 
const globalstack = new GlobalStack(app,'DynamoDBGlobalStack', {
  env: {region: 'us-west-2'}}); appRegions.forEach(function (item, index) { 
  new AppStack(app, 'AppStack-'.concat(item), { 
    env: {account: process.env.CDK_DEFAULT_ACCOUNT, region: item}, 
    globalTableName: globalstack.globalTable.tableName }); 
});

Region References — Mocks

You might have noticed that I am passing only the table name to the app stacks, rather than the actual table object which would be best practice. This is because the actual table is specifically referencing our primary table in us-west-2. I’m afraid in our App stack we will need to either mock out the table OR use the AWS SDK to retrieve the full details.

For most use cases, simply mocking out the table should be all that you need.

import * as cdk from '@aws-cdk/core'; 
import * as dynamodb from '@aws-cdk/aws-dynamodb'; interface CustomStackProps extends cdk.StackProps { 
  readonly globalTableName: string;
  readonly env: any; 
} export class AppStack extends cdk.Stack { 
  constructor(
    scope: cdk.Construct, 
    id: string, 
    props: CustomStackProps) {     super(scope, id, props); 
    const globalTable = new dynamodb.Table.fromTableName(
      this, 'globalTable', props.globalTableName);

We can reference the globalTable to grant IAM permissions etc.

Triggers

I did find that there is ONE instance where there was the need to use the AWS SDK, and that was around implementing triggers off the DynamoDB table. Global Tables are kept in sync by the use of DyanmoDB Streams, and AWS automatically names these streams in the following format:

arn:aws:dynamodb:REGION:AWS-ACCOUNT:table/TABLE-NAME/stream/2021-01-16T19:47:47.531

This timestamp is specific to each region, and so the only way to retrieve this information is to describe the specific region table.

... 
constructor(
  scope: cdk.Construct, 
  id: string, 
  props: CustomStackProps) {   super(scope, id, props); 
  const client = new DynamoDB({ region: props.env.region });   // Query the regions table 
  let globalTableInfoRequest = async() => await client.describeTable({ TableName: props.globalTableName});   globalTableInfoRequest().then( globalTableInfoResult => {     // Mock the table with the specific ARN and Stream ARN 
    const globalTable = dynamodb.Table.fromTableAttributes(
      this, "globalTable", { 
       tableArn: globalTableInfoResult?.Table?.TableArn,
       tableStreamArn: globalTableInfoResult?.Table?.LatestStreamArn 
      });     // Lambda 
    const triggerLambda = new lambda.Function(
      this, 'triggerLambda', { 
        ... 
        environment: { 
          TABLE_NAME: props.globalTableName, 
          ... 
        }
      });     // Grant read access 
    globalTable.grantStreamRead(triggerLambda);     // Deadletter queue 
    const triggerDLQueue = new sqs.Queue(this, 'triggerDLQueue');     // Trigger Event 
    triggerLambda.addEventSource(
      new DynamoEventSource(
        globalTable, { 
        startingPosition: lambda.StartingPosition.TRIM_HORIZON,
        batchSize: 5, 
        bisectBatchOnError: true, 
        onFailure: new SqsDlq(triggerDLQueue), 
        retryAttempts: 10 })); 
    });
  }
}

Regions Available

A final piece to consider is that DynamoDB Global tables in most, but NOT all regions — specifically the following regions do not support them at the time I write this:

Africa (Cape Town) — af-south-1
Asia Pacific (Hong Kong) — ap-east-1
Europe (Milan) — eu-south-1
Middle East (Bahrain) — me-south-1

These four regions are disabled by default, and you will need to enable them if you wanted to use them.

Conclusion

DynamoDB Global tables are a great method to quickly implement a multi-region, multi-master data solution that is managed by AWS. In 2019 the pricing model for Global tables was updated that removed the cost to replicate data between regions, which really elevated this into a great solution.

My next post will examine the DNS routing policy that we will want to implement to ensure users are not impacted by any region downtime in the future.

All code for this post can be accessed on GitHub

Originally published at https://gizmo.codes.