'%20fill='%23000000'%20stroke='none'%3e%3cpath%20d='M%201784.53%205631.9%20C%201775.53%205618.9%201692.53%205479.9%201598.53%205322.9%20C%201395.53%204983.9%201355.53%204919.9%201342.53%204915.9%20C%201332.53%204912.9%201315.53%204938.9%201090.53%205311.9%20C%201023.53%205423.9%20960.53%205523.9%20951.53%205534.9%20C%20936.53%205549.9%20918.53%205552.9%20816.53%205552.9%20C%20751.53%205552.9%20696.53%205548.9%20692.53%205542.9%20C%20689.53%205537.9%20721.53%205475.9%20763.53%205405.9%20C%20806.53%205335.9%20855.53%205252.9%20872.53%205222.9%20C%20889.53%205192.9%20941.53%205106.9%20987.53%205032.9%20C%201032.53%204958.9%201161.53%204746.9%201272.53%204562.9%20C%201383.53%204378.9%201522.53%204147.9%201581.53%204050.9%20L%201689.53%203872.9%20L%201823.53%203872.9%20C%201897.53%203872.9%201957.53%203874.9%201957.53%203876.9%20C%201957.53%203879.9%201887.53%203997.9%201800.53%204139.9%20C%201513.53%204612.9%201487.53%204657.9%201487.53%204674.9%20C%201487.53%204697.9%201788.53%205192.9%201802.53%205192.9%20C%201807.53%205192.9%201830.53%205162.9%201852.53%205125.9%20C%201874.53%205088.9%201902.53%205042.9%201913.53%205023.9%20C%201925.53%205003.9%201993.53%204890.9%202065.53%204772.9%20C%202395.53%204228.9%202458.53%204121.9%202452.53%204112.9%20C%202448.53%204106.9%202382.53%204102.9%202287.53%204102.9%20C%202200.53%204102.9%202126.53%204098.9%202123.53%204094.9%20C%202120.53%204089.9%202147.53%204038.9%202183.53%203979.9%20L%202249.53%203872.9%20L%202563.53%203872.9%20C%202736.53%203872.9%202877.53%203875.9%202877.53%203878.9%20C%202877.53%203882.9%202812.53%203991.9%202734.53%204121.9%20C%202537.53%204445.9%202057.53%205239.9%201929.53%205452.9%20C%201801.53%205664.9%201804.53%205662.9%201784.53%205631.9%20Z'%20style='fill:%20rgb(215,%2032,%2038);'/%3e%3cpath%20d='M%202191.53%205539.9%20C%202187.53%205527.9%202443.53%205099.9%203052.53%204100.9%20L%203191.53%203872.9%20L%203324.53%203872.9%20C%203397.53%203872.9%203457.53%203874.9%203457.53%203876.9%20C%203457.53%203879.9%203366.53%204031.9%203255.53%204214.9%20C%203144.53%204397.9%202964.53%204696.9%202854.53%204877.9%20C%202744.53%205059.9%202607.53%205285.9%202550.53%205380.9%20L%202445.53%205552.9%20L%202321.53%205552.9%20C%202226.53%205552.9%202195.53%205549.9%202191.53%205539.9%20Z'%20style='fill:%20rgb(215,%2032,%2038);'/%3e%3cpath%20d='M%201029.53%204386.9%20C%201014.53%204360.9%20993.53%204325.9%20982.53%204308.9%20C%20970.53%204291.9%20938.53%204237.9%20909.53%204187.9%20C%20880.53%204138.9%20829.53%204053.9%20797.53%203999.9%20C%20764.53%203944.9%20737.53%203894.9%20737.53%203886.9%20C%20737.53%203874.9%20759.53%203872.9%20864.53%203874.9%20L%20991.53%203877.9%20L%201099.53%204049.9%20L%201207.53%204220.9%20L%201144.53%204326.9%20C%201110.53%204384.9%201076.53%204432.9%201069.53%204432.9%20C%201062.53%204432.9%201044.53%204411.9%201029.53%204386.9%20Z'%20style='fill:%20rgb(215,%2032,%2038);'/%3e%3c/g%3e%3c/svg%3e){kind=small}
Project Overview
Baanx, a leading crypto-finance platform enabling users to buy, spend, and borrow against digital assets, needed to unlock actionable insights from their operational data. Their previous data management investments hadn't delivered—query applications couldn't extract insights efficiently, and analytics remained inaccessible. We designed and delivered an end-to-end ETL pipeline using AWS Glue to synchronise data from AWS RDS to Amazon Redshift, enabling powerful analytics and anomaly detection through Tableau.
The Challenge
Baanx had invested in data management but the outcomes fell short. Their existing setup couldn't extract actionable insights in an efficient or simple manner. Data was siloed in AWS RDS with no pathway to analytical tooling, preventing the team from running the queries needed for business reporting, anomaly detection, and trend analysis.
- Existing data infrastructure failed to meet business reporting requirements
- No efficient pathway from operational databases to analytical tools like Tableau
- Inability to detect anomalies or perform trend analysis on transactional data
- Manual data processes creating bottlenecks in decision-making
Solution Architecture
We implemented a robust ETL pipeline using AWS Glue as the core orchestration engine. The solution extracts data from AWS RDS, transforms it for analytical querying, and loads it into Amazon Redshift—making the data immediately accessible via Tableau for dashboards and ad-hoc analysis.
- AWS Glue Crawlers to populate the Data Catalog with metadata table definitions
- Custom AWS Glue scripts for data transformation and schema mapping
- Scheduled batch synchronisation with manual trigger capability for on-demand refreshes
- Data modelling optimised for simple querying through Tableau and other BI tools
Technical Implementation
The project was delivered in three focused milestones. First, we set up the infrastructure, tooling, IAM permissions, and Glue profiles. Next, we built the synchronisation layer with scheduled batch jobs and manual trigger support. Finally, we transformed the data models to support straightforward querying in Tableau, ensuring analysts could self-serve without engineering support.
- Infrastructure setup: AWS Glue jobs, IAM roles, S3 staging buckets, and Redshift cluster configuration—all provisioned and managed through Terraform for reproducible, version-controlled infrastructure
- Batch sync engine with configurable schedules and on-demand manual triggers
- Data transformation layer converting RDS schemas into Redshift-optimised star schemas
- Extended scope planning for change data capture (CDC) trails and computed business metrics
Technologies
Results & Impact
- Delivered the complete pipeline in under 3 weeks across three milestones
- Enabled self-service analytics through Tableau for business and operations teams
- Scheduled and on-demand data sync eliminated manual data extraction processes
- Data transformation layer reduced query complexity, making insights accessible to non-technical stakeholders
- Laid groundwork for extended scope: historical change tracking and computed business metrics
- Provided Baanx with anomaly detection capabilities across their crypto-finance transaction data
Why It Matters
By building a clean, reliable data pipeline from RDS to Redshift, we gave Baanx the analytical foundation they'd been missing. Their teams can now query transaction data, detect anomalies, and generate business reports through Tableau without engineering intervention. The modular architecture also positions them for the extended scope—historical data trails and computed metrics—as their analytics needs evolve.
Services Behind This Project
The same senior teams that delivered this work: