Senior Data Engineer

1. Execute 

• Collaborate with data product managers to gather data requirement 

• Execute ETL solutions including data security, data quality and performance requirements.  

• Prepare documentation of data product lineage and all other related ETL topics. 

2. Data Extraction, Load and Transformation

• Implement and maintain ELT pipelines to efficiently ingest and transform data from a wide variety of data sources and deliver datasets that meet business requirements. 
• Ensure efficient and reliable data mapping to support business needs.
• Deliver complete documentation and knowledge transfer sessions for the Team and business partners.
• Maintain existing solutions, implement optimizations and enhancements, monitor data quality.
• Support the development and maintenance of scalable data pipelines leveraging Azure Synapse, PySpark, APIs, and SQL & performing advanced data cleaning, transformation, and manipulation to ensure high-quality, and reliable data flows.  

3. Process Improvement, Performance and Cost optimization tuning

• Collaborate with Data Science, AI, and Data product teams to optimize performance and cost effectiveness of their solutions. 
• Identify and support the design of internal process improvements, including automating manual processes, optimizing data product delivery, and redesigning solutions for enhanced scalability. 
• Implement solution adjustments to improve performance and cost-effectiveness of data products.

4. Issue Resolution and Support

• Assist stakeholders with data-related product pipeline issues and support their data product needs. 
• Work with the Analytics Operational Support team to investigate, troubleshoot, and resolve data errors / discrepancies.