Data Engineering

ETL Optimization

Project by Sirajuddin Mohammed

The business problem and approach

Refactored and optimized traditional ETL processes by migrating from Python to PySpark on an HDInsight cluster, loading data efficiently into Azure PolyBase tables. Developed automated tracking and alerting with Talend, significantly increasing throughput and enabling scalable, high-speed data processing.

Implementation highlights

Automated monitoring and alerting integrated with PySpark jobs using Talend, Improved throughput by saving 15 hours daily via PySpark and Azure PolyBase, Optimized ETL leveraging PySpark parallelism and scalable PolyBase storage

Delivery and outcomes

15hours saved daily in data processing, 40% increase in ETL job speed

Technologies used

PySpark
Azure HDInsight
Azure PolyBase Tables
Talend
Python

Have a related business challenge?

Explore data science and forecasting consulting, or share what you would like to improve.

Discuss your business challenge

← Back to selected work