I am curious what is issue with existing migration tools? There are so many tools out there. What is the problem they are not solving?

Hi! Most migration tools I have seen simply run the SQL needed to perform the update directly. This causes trouble during deploys as there will inevitably be some downtime between applying the migration and rolling out the application changes which work with the new schema.

Say for example you want to rename a column for a running application. With a regular migration tool, you can't just run the migration as the old application deployment still expects the old column name, changing it breaks the application. Instead you might do something like this:

1. Bring down the old application instances

2. Apply the migration

3. Deploy new application instances which can work with the new schema

All of this is downtime which might not be too bad if deployments are quick and the migration doesn't take much time. For more complex migrations that take a longer time, like changing data in a bunch of rows, this downtime can become significant. The way I've seen that handled is that developers have to manually perform several deployments, for example:

1. Add temporary column to table with no data. Update application to fill it for new rows. Deploy.

2. Write batch job to backfill old rows. Deploy. Wait for job to finish.

3. Update application to use new column. Deploy.

4. Remove old column. Deploy.

This is a lot of manual work which Reshape aims to automate, hopefully reducing the burden on developers and the risk for human errors.

Sounds neat! We currently have to deploy 2-3 releases to just have 1 logical migration and it's pretty error-prone and complicates our release schedule. We basically gave up trying to rename columns at this point because it's too much effort for little worth, but it confuses the hell out of new devs when a column name in a DB doesn't match the name in the code.

Your article mentions that it allows to avoid locking tables during migrations and it's something we haven't solved yet either - in one of the recent migrations that touched a very fat table we had to write a custom migration which applied changes in batches to avoid locking the entire table under one fat transaction.

Our DB is sharded: we first apply migrations on thousands of DBs and only then deploy code. Such migrations can take up to an hour to finish before code is deployed, so we have to make sure our migrations are forward-compatible because old code can see new schemas for a prolonged time. Will Reshape work well for this mechanism? Sometimes we have to run custom migrations which call code, I guess Reshape can't replace it because it's strictly DB-based?

Unfortunately, we use MySQL, not PostreSQL, so I wonder, if it's portable to other DBs.

I migrated huge tables with absolutely no impact on production