Migrating an RDS DB instance to an Aurora DB cluster with CloudFormation

There are several guides describing how to migrate a plain RDS instance to an Aurora cluster using read replica. But none of them describes how the new Aurora cluster can afterwards be maintained further with CloudFormation.

This guide describes a migration path I've used on production services. A few manual steps and a short downtime - or at least read-only mode - are required, but the final cluster will then be part of an already existing CloudFormation stack.

High-Level Process

With or without CloudFormation, the high-level steps are the following

1. Create an Aurora read-replica of the existing database
2. Stop writing to the source DB instance
3. Promote the read-replica to a stand-alone cluster
4. Use the Aurora cluster from the application

Background

Read-Replica and CloudFormation

Read-replicas are defined in CloudFormation on a DBCluster resource by setting the ReplicationSourceIdentifier property. The value is the ARN of the source DB instance. When this property is set, the DBCluster must not have the properties MasterUsername, MasterUserPassword and DatabaseName, because their values are taken from the replication source.

After deployment, the replication from the source DB instance to the Aurora replica starts automatically.

Promotion to Stand-Alone

Read-replica done, now it has to be promoted to a stand-alone DB. You might now think - just as I did - that removing the ReplicationSourceIdentifier and setting the other 3 properties would just do that.

WRONG! A change of ReplicationSourceIdentifier replaces the cluster with full data loss. Promotion is just not possible with a CloudFormation stack-update.

Migraton Path

The read-replica can be defined in CloudFormation, and also the final stand-alone cluster, but promotion has to be done manually.

Playbook

1. Create the read-replica with CloudFormation
   • Set DeletionPolicy to Retain on the DBCluster and the cluster's DBInstance. This will preserve the resources even when they are deleted from the stack.
2. Detach the read-replica from the CloudFormation stack
   • Remove the DBCluster and DBInstance from the CloudFormation template and deploy it. This will remove the resources from the stack, but due to the deletion policy, they are not removed.
3. Promote the replica to a stand-alone cluster manually
   • Stop writing to the source database by either shutting down the application or setting it to read-only mode
   • Wait for a few minutes. Latest when write operations are down to zero, the replication is done.
   • Promote the Aurora cluster using the AWS console. It's done when the Replication Source is no longer visible on the configuration tab.
4. Import the stand-alone cluster into the CloudFormation stack
   • Add the DBCluster and DBInstance resources back to the stack, remove the ReplicationSourceIdentifier and add MasterUsername, MasterUserPassword and DatabaseName. But don't deploy yet.
   • Open the stack in the AWS console and execute Import using the modified template. It will show you the cluster and the DB instance as missing resources and ask for their identifiers. The identifier of the cluster is the value of DBClusterIdentifier and the DB instance identifier is taken from DBInstanceIdentifier.
   • After the import, the stand-alone cluster is part of the stack and can from then on be maintained using the CloudFormation template.