Platforms with fast data migration
Introduction
Transferring information when changing or updating the platform is a critical task: accounting for balances, history of bets, bonuses, KYC data and campaign settings cannot be lost or distorted. Modern solutions use automated ETL pipelines and Change Data Capture (CDC) to complete the migration in hours or even minutes without business downtime.
1. Classification of migrations
1. Cold migration
Full export-import, requires platform shutdown.
Suitable for low activity or planned maintenance window.
2. Hot migration
Parallel chạy ETL + CDC replication, cut-over in seconds.
Suitable for large operators with round-the-clock traffic.
2. ETL and CDC Architecture
```mermaid
flowchart LR
subgraph Source
DB1[(Old DB)]
Stream1[(Old DB CDC)]
end
subgraph Conveyor
ETL[ETL Job]
CDC[CDC Consumer]
Validator[Data Validator]
end
subgraph Purpose
DB2[(New DB)]
end
ETL Job: once a night or on a schedule, reads a full dump of tables, transforms formats and loads into a new scheme.
CDC Consumer: listens to WAL logs (Debezium/MySQL Binlog), skips INSERT/UPDATE/DELETE in near-real-time mode.
Validator: verifies checksums and counters of records after base load and during streaming replication.
3. Migration stages
1. Analysis and mapping (1-2 days)
Comparison of schemes of the old and new database, determination of field correspondences (for example, 'player _ balance' → 'wallet. real_balance`).
Definition of type conversions: rows → JSON, timestamps, ENUM → reference tables.
2. Preparation of test environment (1-2 days)
Deployment of a staging cluster with a voluminous snapshot of production data.
Configure ETL and CDC connectors on test data.
3. "cold load" (2-4 hours)
Exporting a full dump from source DB → parallel import to target DB.
Disabling non-duplicated processes (for example, a bonus engine) at boot time.
4. Start CDC replication (continuous)
Start listening for changes from when ETL loading started.
Accumulation of the "tail" of operations until the cut-over is ready.
5. Cut-over and traffic switching (1-5 minutes)
Temporarily stopping applications to align the remainder of the CDC tail.
Reconfiguring connection strings to a new database.
Smoke tests of basic scripts (login, deposit, spin, within).
6. Validation and rollback (1-2 hours)
Checksum check for key tables - users, balances, transaction history.
If critical mismatches - automatic rollback to snapshot.
4. Testing and validation
Row counts & checksums: comparison of the number of records and hashes by tables.
Domain tests: sample scenarios - betting, bonus and withdrawal operations.
End-to-End tests: Automated Cypress/Playwright scripts run key flow in staging after migration.
5. Minimizing downtime
Blue-Green Database
Parallel database instances...
Proxy-level Cut-over
Using a proxy (PgBouncer) for a smooth switchover with an incoming connection queue.
Feature Flags
Disabling part of the functionality during the migration, so as not to completely block all services.
6. Tools and platforms
Debezium + Kafka для CDC с MySQL/PostgreSQL.
Airbyte, Fivetran, Talend for ETL pipelines.
Flyway/Liquibase for schema migrations and database versioning.
HashiCorp Vault for safe storage credentials and rotation.
Conclusion
Platforms with support for fast data migrations build a process around a combination of ETL loading and CDC replication, rigorous testing and validation checks. With a competent architecture and automation, downtime is reduced to a few minutes, and the risk of data loss or mismatch is zero.
Transferring information when changing or updating the platform is a critical task: accounting for balances, history of bets, bonuses, KYC data and campaign settings cannot be lost or distorted. Modern solutions use automated ETL pipelines and Change Data Capture (CDC) to complete the migration in hours or even minutes without business downtime.
1. Classification of migrations
1. Cold migration
Full export-import, requires platform shutdown.
Suitable for low activity or planned maintenance window.
2. Hot migration
Parallel chạy ETL + CDC replication, cut-over in seconds.
Suitable for large operators with round-the-clock traffic.
2. ETL and CDC Architecture
```mermaid
flowchart LR
subgraph Source
DB1[(Old DB)]
Stream1[(Old DB CDC)]
end
subgraph Conveyor
ETL[ETL Job]
CDC[CDC Consumer]
Validator[Data Validator]
end
subgraph Purpose
DB2[(New DB)]
end
| DB1 --> | full dump | ETL --> Validator --> DB2 |
|---|---|---|
| Stream1 --> | real-time changes | CDC --> Validator --> DB2 |
| ``` |
ETL Job: once a night or on a schedule, reads a full dump of tables, transforms formats and loads into a new scheme.
CDC Consumer: listens to WAL logs (Debezium/MySQL Binlog), skips INSERT/UPDATE/DELETE in near-real-time mode.
Validator: verifies checksums and counters of records after base load and during streaming replication.
3. Migration stages
1. Analysis and mapping (1-2 days)
Comparison of schemes of the old and new database, determination of field correspondences (for example, 'player _ balance' → 'wallet. real_balance`).
Definition of type conversions: rows → JSON, timestamps, ENUM → reference tables.
2. Preparation of test environment (1-2 days)
Deployment of a staging cluster with a voluminous snapshot of production data.
Configure ETL and CDC connectors on test data.
3. "cold load" (2-4 hours)
Exporting a full dump from source DB → parallel import to target DB.
Disabling non-duplicated processes (for example, a bonus engine) at boot time.
4. Start CDC replication (continuous)
Start listening for changes from when ETL loading started.
Accumulation of the "tail" of operations until the cut-over is ready.
5. Cut-over and traffic switching (1-5 minutes)
Temporarily stopping applications to align the remainder of the CDC tail.
Reconfiguring connection strings to a new database.
Smoke tests of basic scripts (login, deposit, spin, within).
6. Validation and rollback (1-2 hours)
Checksum check for key tables - users, balances, transaction history.
If critical mismatches - automatic rollback to snapshot.
4. Testing and validation
Row counts & checksums: comparison of the number of records and hashes by tables.
Domain tests: sample scenarios - betting, bonus and withdrawal operations.
End-to-End tests: Automated Cypress/Playwright scripts run key flow in staging after migration.
5. Minimizing downtime
Blue-Green Database
Parallel database instances...
Proxy-level Cut-over
Using a proxy (PgBouncer) for a smooth switchover with an incoming connection queue.
Feature Flags
Disabling part of the functionality during the migration, so as not to completely block all services.
6. Tools and platforms
Debezium + Kafka для CDC с MySQL/PostgreSQL.
Airbyte, Fivetran, Talend for ETL pipelines.
Flyway/Liquibase for schema migrations and database versioning.
HashiCorp Vault for safe storage credentials and rotation.
Conclusion
Platforms with support for fast data migrations build a process around a combination of ETL loading and CDC replication, rigorous testing and validation checks. With a competent architecture and automation, downtime is reduced to a few minutes, and the risk of data loss or mismatch is zero.
